[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:45:49 dist-test-slave-6tjh krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:45:49 dist-test-slave-6tjh krb5kdc[395](info): set up 2 sockets
Sep 29 02:45:49 dist-test-slave-6tjh krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 02:45:50.630062 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:45:50 dist-test-slave-6tjh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:45:50.806444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44367
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:51.141264 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:51.141553 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:51.141641 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:51.148862 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:45:51.148973 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:51.149019 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:51.149066 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:45:51.149111 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:45:51.159243 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:51.160471 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:51.161778 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:51.173506 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:51.173473 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:51.173563 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:51.174026 408 server_base.cc:1034] running on GCE node
I20260929 02:45:51.175261 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:51.176728 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:51.177943 408 hybrid_clock.cc:648] HybridClock initialized: now 1790649951177849 us; error 104 us; skew 500 ppm
I20260929 02:45:51.178279 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:51 dist-test-slave-6tjh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:51.181315 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:45:51.182821 408 webserver.cc:460] Webserver started at http://127.0.97.62:33763/ using document root <none> and password file <none>
I20260929 02:45:51.183485 408 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:51.183605 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:51.183899 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:51.186815 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "9944efc5ee0f45b2bd0376076786bf4e"
format_stamp: "Formatted at 2026-09-29 02:45:51 on dist-test-slave-6tjh"
server_key: "34dae759d637c512fc418cd1202edb30"
server_key_iv: "38d71b260dd3d389b15fb72d1de14c9c"
server_key_version: "encryptionkey@0"
I20260929 02:45:51.187554 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "9944efc5ee0f45b2bd0376076786bf4e"
format_stamp: "Formatted at 2026-09-29 02:45:51 on dist-test-slave-6tjh"
server_key: "34dae759d637c512fc418cd1202edb30"
server_key_iv: "38d71b260dd3d389b15fb72d1de14c9c"
server_key_version: "encryptionkey@0"
I20260929 02:45:51.192699 408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:45:51.195998 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:51.197376 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:45:51.197541 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "9944efc5ee0f45b2bd0376076786bf4e"
format_stamp: "Formatted at 2026-09-29 02:45:51 on dist-test-slave-6tjh"
server_key: "34dae759d637c512fc418cd1202edb30"
server_key_iv: "38d71b260dd3d389b15fb72d1de14c9c"
server_key_version: "encryptionkey@0"
I20260929 02:45:51.197724 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:51.223984 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:51.228018 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:51.228325 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:51.248010 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44367
I20260929 02:45:51.248039 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44367 every 8 connection(s)
I20260929 02:45:51.249997 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:45:51.253473 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 408
I20260929 02:45:51.253783 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:45:51.253790 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:45:51.255579 388 external_mini_cluster.cc:1428] Setting key 1ef0cd73fc1def38d66ba6fb0a04f11a
I20260929 02:45:51.264922 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e: Bootstrap starting.
I20260929 02:45:51.267705 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e: Neither blocks nor log segments found. Creating new log.
I20260929 02:45:51.268868 475 log.cc:824] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:45:51.271713 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e: No bootstrap required, opened a new log
I20260929 02:45:51.278759 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944efc5ee0f45b2bd0376076786bf4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44367 } }
I20260929 02:45:51.279111 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:45:51.279219 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9944efc5ee0f45b2bd0376076786bf4e, State: Initialized, Role: FOLLOWER
I20260929 02:45:51.279865 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944efc5ee0f45b2bd0376076786bf4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44367 } }
I20260929 02:45:51.280102 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:45:51.280252 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:45:51.280441 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 0 FOLLOWER]: Advancing to term 1
Sep 29 02:45:51 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:51.283138 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944efc5ee0f45b2bd0376076786bf4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44367 } }
I20260929 02:45:51.283622 475 leader_election.cc:302] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9944efc5ee0f45b2bd0376076786bf4e; no voters:
I20260929 02:45:51.284221 475 leader_election.cc:288] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:45:51.284422 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:45:51.285487 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [term 1 LEADER]: Becoming Leader. State: Replica: 9944efc5ee0f45b2bd0376076786bf4e, State: Running, Role: LEADER
I20260929 02:45:51.286237 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944efc5ee0f45b2bd0376076786bf4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44367 } }
I20260929 02:45:51.286473 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:45:51.288952 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9944efc5ee0f45b2bd0376076786bf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944efc5ee0f45b2bd0376076786bf4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44367 } } }
I20260929 02:45:51.289719 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [sys.catalog]: This master's current role is: LEADER
I20260929 02:45:51.294339 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9944efc5ee0f45b2bd0376076786bf4e. Latest consensus state: current_term: 1 leader_uuid: "9944efc5ee0f45b2bd0376076786bf4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9944efc5ee0f45b2bd0376076786bf4e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44367 } } }
I20260929 02:45:51.294742 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e [sys.catalog]: This master's current role is: LEADER
I20260929 02:45:51.298828 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:51.259224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40308 (local address 127.0.97.62:44367)
0929 02:45:51.259963 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:51.259973 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:51.261180 (+ 1207us) server_negotiation.cc:407] Connection header received
0929 02:45:51.270070 (+ 8890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:51.270096 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:51.270504 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:51.270883 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:51.274155 (+ 3272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:51.275527 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:51.277234 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:51.277906 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:51.278118 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:51.283896 (+ 5778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:51.283905 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:51.283921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:51.283959 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:51.288988 (+ 5029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:51.290433 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:51.290444 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:51.290455 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:51.290726 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:51.291453 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:51.291460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:51.291466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:51.292338 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:51.296163 (+ 3825us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:51.296497 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 02:45:51.297077 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":349,"threads_started":1}
I20260929 02:45:51.303110 485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:45:51.311491 485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:45:51.320801 485 catalog_manager.cc:1347] Generated new cluster ID: b43332d5f08d4c90844a5f00ab5188c4
I20260929 02:45:51.320935 485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:45:51.341483 485 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:45:51.343428 485 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:45:51.360168 485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9944efc5ee0f45b2bd0376076786bf4e: Generated new TSK 0
I20260929 02:45:51.361347 485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:45:51.405364 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44367
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:51.746256 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:51.746538 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:51.746702 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:51.754374 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:51.754478 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:51.754683 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:45:51.765394 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:51.766738 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:51.768119 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:51.782251 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:51.782218 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:51.783289 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:51.783269 501 server_base.cc:1034] running on GCE node
I20260929 02:45:51.784027 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:51.785061 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:51.786294 501 hybrid_clock.cc:648] HybridClock initialized: now 1790649951786248 us; error 43 us; skew 500 ppm
I20260929 02:45:51.786674 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:51 dist-test-slave-6tjh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:51.790082 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:45:51.791591 501 webserver.cc:460] Webserver started at http://127.0.97.1:46057/ using document root <none> and password file <none>
I20260929 02:45:51.792322 501 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:51.792464 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:51.792820 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:51.796166 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "5061c74198384cc7aa078de74d4558c1"
format_stamp: "Formatted at 2026-09-29 02:45:51 on dist-test-slave-6tjh"
server_key: "ce1bd5c4cf718069eb8faf5f985f9b11"
server_key_iv: "f8447e6d6d6d2ba24596f246aa817190"
server_key_version: "encryptionkey@0"
I20260929 02:45:51.796998 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5061c74198384cc7aa078de74d4558c1"
format_stamp: "Formatted at 2026-09-29 02:45:51 on dist-test-slave-6tjh"
server_key: "ce1bd5c4cf718069eb8faf5f985f9b11"
server_key_iv: "f8447e6d6d6d2ba24596f246aa817190"
server_key_version: "encryptionkey@0"
I20260929 02:45:51.803107 501 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:45:51.806602 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:51.808044 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:45:51.808262 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "5061c74198384cc7aa078de74d4558c1"
format_stamp: "Formatted at 2026-09-29 02:45:51 on dist-test-slave-6tjh"
server_key: "ce1bd5c4cf718069eb8faf5f985f9b11"
server_key_iv: "f8447e6d6d6d2ba24596f246aa817190"
server_key_version: "encryptionkey@0"
I20260929 02:45:51.808491 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:51.826931 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:51.831794 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:51.832113 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:51.833133 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:45:51.835016 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:45:51.835145 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:51.835310 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:45:51.835379 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:51.872507 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42361
I20260929 02:45:51.872536 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42361 every 8 connection(s)
I20260929 02:45:51.873991 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:45:51.883347 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 501
I20260929 02:45:51.883585 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:45:51.884300 388 external_mini_cluster.cc:1428] Setting key e431ffeee55baa43c1a58575b275b13b
Sep 29 02:45:51 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:51.897779 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:51.877135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44487 (local address 127.0.97.62:44367)
0929 02:45:51.877545 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:51.877550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:51.878648 (+ 1098us) server_negotiation.cc:407] Connection header received
0929 02:45:51.880537 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:51.880565 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:51.880664 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:51.880789 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:51.882314 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:51.883033 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:51.884819 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:51.885374 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:51.885504 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:51.890303 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:51.890313 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:51.890321 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:51.890366 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:51.894015 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:51.895109 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:51.895113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:51.895116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:51.895258 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:51.895815 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:51.895819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:51.895821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:51.896142 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:51.896266 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:51.897321 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0929 02:45:51.897502 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":211,"threads_started":1}
I20260929 02:45:51.899598 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:51.877769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44367 (local address 127.0.97.1:44487)
0929 02:45:51.878482 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:51.878520 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:51.880184 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:51.881027 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:51.881036 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:51.881564 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:45:51.882136 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:51.882153 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:51.883203 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:51.883207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:51.884677 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:51.884688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:51.885570 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:51.885575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:51.885785 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:51.886557 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:45:51.886586 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:45:51.890076 (+ 3490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:45:51.894191 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:51.894205 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:51.894227 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:51.894942 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:51.895391 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:51.895399 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:51.895407 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:51.895681 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:51.896286 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:45:51.896298 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:45:51.896852 (+ 554us) client_negotiation.cc:769] Sending connection context
0929 02:45:51.897225 (+ 373us) client_negotiation.cc:240] Negotiation successful
0929 02:45:51.897875 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":290,"threads_started":1}
I20260929 02:45:51.901736 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44367
I20260929 02:45:51.902076 629 heartbeater.cc:464] Registering TS with master...
I20260929 02:45:51.902971 629 heartbeater.cc:511] Master 127.0.97.62:44367 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:45:51.905426 440 ts_manager.cc:194] Registered new tserver with Master: 5061c74198384cc7aa078de74d4558c1 (127.0.97.1:42361)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:45:51.906998 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:44487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:45:51.926137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44367
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:52.274960 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:52.275492 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:52.275733 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:52.285589 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:52.285719 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:52.285866 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:45:52.296342 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:52.297672 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:52.298949 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:52.312882 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:52.312903 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:52.312978 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:52.318444 635 server_base.cc:1034] running on GCE node
I20260929 02:45:52.319152 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:52.320147 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:52.321448 635 hybrid_clock.cc:648] HybridClock initialized: now 1790649952321376 us; error 65 us; skew 500 ppm
I20260929 02:45:52.321856 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:52 dist-test-slave-6tjh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:52.325691 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:45:52.327271 635 webserver.cc:460] Webserver started at http://127.0.97.2:39901/ using document root <none> and password file <none>
I20260929 02:45:52.327971 635 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:52.328074 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:52.328327 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:52.331391 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "0dbe7a3e71594cea921a2db5d288e706"
format_stamp: "Formatted at 2026-09-29 02:45:52 on dist-test-slave-6tjh"
server_key: "80eb208346163c1356ea1e1cfe0cb108"
server_key_iv: "c2816aac55b14629a869284e7f25ad55"
server_key_version: "encryptionkey@0"
I20260929 02:45:52.332176 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "0dbe7a3e71594cea921a2db5d288e706"
format_stamp: "Formatted at 2026-09-29 02:45:52 on dist-test-slave-6tjh"
server_key: "80eb208346163c1356ea1e1cfe0cb108"
server_key_iv: "c2816aac55b14629a869284e7f25ad55"
server_key_version: "encryptionkey@0"
I20260929 02:45:52.337730 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:45:52.341499 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:52.343106 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:45:52.343303 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "0dbe7a3e71594cea921a2db5d288e706"
format_stamp: "Formatted at 2026-09-29 02:45:52 on dist-test-slave-6tjh"
server_key: "80eb208346163c1356ea1e1cfe0cb108"
server_key_iv: "c2816aac55b14629a869284e7f25ad55"
server_key_version: "encryptionkey@0"
I20260929 02:45:52.343520 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:52.358342 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:52.363440 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:52.363768 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:52.364881 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:45:52.366357 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:45:52.366431 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:52.366570 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:45:52.366626 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:52.403625 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42027
I20260929 02:45:52.403680 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42027 every 8 connection(s)
I20260929 02:45:52.405073 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:45:52.413465 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 635
I20260929 02:45:52.413655 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:45:52.414016 388 external_mini_cluster.cc:1428] Setting key aac10aa96c3c16397cc03436d4269b22
Sep 29 02:45:52 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:52.428135 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:52.407808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57071 (local address 127.0.97.62:44367)
0929 02:45:52.408238 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:52.408244 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:52.409736 (+ 1492us) server_negotiation.cc:407] Connection header received
0929 02:45:52.411499 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:52.411503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:52.411575 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:52.411672 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:52.413304 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.414095 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.416020 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.416440 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.416533 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:52.421331 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:52.421336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:52.421340 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:52.421370 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:52.424810 (+ 3440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:52.425512 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:52.425520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:52.425526 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:52.425684 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:52.426101 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:52.426109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:52.426115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:52.426499 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:52.426653 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:52.427377 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 02:45:52.427626 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":206,"threads_started":1}
I20260929 02:45:52.428821 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:52.408419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44367 (local address 127.0.97.2:57071)
0929 02:45:52.409550 (+ 1131us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:52.409587 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:52.411241 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:52.411936 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:52.411945 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:52.412467 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:45:52.413126 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:52.413149 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.414251 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.414257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:52.415877 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:52.415891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.416696 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.416705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:52.417038 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:52.417775 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:45:52.417809 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:45:52.421093 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:45:52.424943 (+ 3850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:52.424950 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:52.424963 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:52.425379 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:52.425801 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:52.425804 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:52.425806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:52.425984 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:52.426619 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:45:52.426625 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:45:52.426971 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 02:45:52.427202 (+ 231us) client_negotiation.cc:240] Negotiation successful
0929 02:45:52.427605 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"thread_start_us":716,"threads_started":1}
I20260929 02:45:52.430848 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44367
I20260929 02:45:52.431203 763 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:45:52.432137 763 heartbeater.cc:511] Master 127.0.97.62:44367 requested a full tablet report, sending...
I20260929 02:45:52.433841 440 ts_manager.cc:194] Registered new tserver with Master: 0dbe7a3e71594cea921a2db5d288e706 (127.0.97.2:42027)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 02:45:52.434795 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:45:52.456038 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44367
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:52.785204 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:52.785487 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:52.785583 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:52.793633 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:52.793776 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:52.793900 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:45:52.804904 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44367
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:52.806198 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:52.807541 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:52.821028 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:52.821635 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:52.821671 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:52.822613 769 server_base.cc:1034] running on GCE node
I20260929 02:45:52.823303 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:52.824326 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:52.825562 769 hybrid_clock.cc:648] HybridClock initialized: now 1790649952825474 us; error 107 us; skew 500 ppm
I20260929 02:45:52.825929 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:52 dist-test-slave-6tjh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:52.832077 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:45:52.833635 769 webserver.cc:460] Webserver started at http://127.0.97.3:40997/ using document root <none> and password file <none>
I20260929 02:45:52.834333 769 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:52.834436 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:52.834697 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:52.837677 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "0c34dc5803b043d7a17412374f825b30"
format_stamp: "Formatted at 2026-09-29 02:45:52 on dist-test-slave-6tjh"
server_key: "f52b242030a60128abdfb61a10529f1f"
server_key_iv: "9b7605bb01640c833ecf022df4718f23"
server_key_version: "encryptionkey@0"
I20260929 02:45:52.838400 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0c34dc5803b043d7a17412374f825b30"
format_stamp: "Formatted at 2026-09-29 02:45:52 on dist-test-slave-6tjh"
server_key: "f52b242030a60128abdfb61a10529f1f"
server_key_iv: "9b7605bb01640c833ecf022df4718f23"
server_key_version: "encryptionkey@0"
I20260929 02:45:52.843878 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:45:52.848044 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:52.849661 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:45:52.849858 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "0c34dc5803b043d7a17412374f825b30"
format_stamp: "Formatted at 2026-09-29 02:45:52 on dist-test-slave-6tjh"
server_key: "f52b242030a60128abdfb61a10529f1f"
server_key_iv: "9b7605bb01640c833ecf022df4718f23"
server_key_version: "encryptionkey@0"
I20260929 02:45:52.850063 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:52.866055 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:52.870296 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:52.870592 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:52.871601 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:45:52.873087 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:45:52.873194 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:52.873332 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:45:52.873401 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:52.910773 629 heartbeater.cc:503] Master 127.0.97.62:44367 was elected leader, sending a full tablet report...
I20260929 02:45:52.920477 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42503
I20260929 02:45:52.920501 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42503 every 8 connection(s)
I20260929 02:45:52.922467 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:45:52.929054 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 769
I20260929 02:45:52.929276 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:45:52.929911 388 external_mini_cluster.cc:1428] Setting key df010e0a1a8c2b0281f59c303a78b535
Sep 29 02:45:52 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:52.947142 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:52.925842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56005 (local address 127.0.97.62:44367)
0929 02:45:52.926083 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:52.926093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:52.927292 (+ 1199us) server_negotiation.cc:407] Connection header received
0929 02:45:52.929897 (+ 2605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:52.929903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:52.930012 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:52.930129 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:52.932026 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.933438 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.935180 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.935937 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.936106 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:52.940869 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:52.940876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:52.940882 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:52.940924 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:52.943567 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:52.944666 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:52.944672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:52.944675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:52.944811 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:52.945298 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:52.945303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:52.945306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:52.945695 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:52.945863 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:52.946569 (+ 706us) server_negotiation.cc:299] Negotiation successful
0929 02:45:52.946798 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:45:52.948249 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:52.926261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44367 (local address 127.0.97.3:56005)
0929 02:45:52.927101 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:52.927152 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:52.929550 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:52.930390 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:52.930398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:52.931041 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:45:52.931824 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:52.931844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.933642 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.933648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:52.935017 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:52.935029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.936222 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.936229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:52.936565 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:52.937425 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:45:52.937461 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:45:52.940577 (+ 3116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:45:52.943740 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:52.943749 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:52.943782 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:52.944227 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:52.944951 (+ 724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:52.944956 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:52.944960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:52.945166 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:52.945847 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:45:52.945855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:45:52.946213 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 02:45:52.946492 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 02:45:52.947046 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":385,"threads_started":1}
I20260929 02:45:52.949754 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44367
I20260929 02:45:52.950032 897 heartbeater.cc:464] Registering TS with master...
I20260929 02:45:52.950693 897 heartbeater.cc:511] Master 127.0.97.62:44367 requested a full tablet report, sending...
I20260929 02:45:52.952040 440 ts_manager.cc:194] Registered new tserver with Master: 0c34dc5803b043d7a17412374f825b30 (127.0.97.3:42503)
I20260929 02:45:52.952945 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56005
I20260929 02:45:52.958948 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:45:52 dist-test-slave-6tjh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649952, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:45:52 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649952, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:45:53.004851 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:52.989042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.0.97.1:42361)
0929 02:45:52.989653 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:52.989661 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:52.989690 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:45:52.989785 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:52.989792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:52.990014 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:52.990141 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:52.991505 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.992662 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.993920 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:52.994499 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:52.994700 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:52.999015 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:52.999025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:52.999040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:52.999072 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:53.002072 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.002760 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.002768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.002770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.002894 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.003264 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.003270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.003273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.003610 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:53.003922 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:53.004204 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 02:45:53.004470 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":321,"threads_started":1}
W20260929 02:45:53.006139 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42814
Sep 29 02:45:53 dist-test-slave-6tjh krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:45:53 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:45:53.051954 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:53.037310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42824 (local address 127.0.97.1:42361)
0929 02:45:53.037583 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:53.037592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:53.037908 (+ 316us) server_negotiation.cc:407] Connection header received
0929 02:45:53.038129 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:53.038134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:53.038195 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:53.038298 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:53.039654 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.040331 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.041606 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.042030 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.042164 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:53.046525 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:53.046531 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:53.046535 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:53.046573 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:53.049531 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.050196 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.050201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.050204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.050331 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.050702 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.050705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.050707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.050962 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:53.051144 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:53.051475 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 02:45:53.051689 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:45:53.053654 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 501
I20260929 02:45:53.066237 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:53.054765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.0.97.1:42361)
0929 02:45:53.054947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:53.054953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:53.055081 (+ 128us) server_negotiation.cc:407] Connection header received
0929 02:45:53.055226 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:53.055229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:53.055282 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:53.055354 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:53.056541 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.057260 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.058534 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.058945 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.059028 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:53.061281 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:53.061285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:53.061289 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:53.061315 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:53.063960 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.064586 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.064595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.064602 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.064763 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.065064 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.065066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.065068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.065294 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:53.065462 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:53.065714 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:45:53.065877 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:45:53.067346 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:53.150415 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 501
I20260929 02:45:53.166188 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 635
Sep 29 02:45:53 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:45:53.182397 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:53.167118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37706 (local address 127.0.97.2:42027)
0929 02:45:53.167670 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:53.167677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:53.167702 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:45:53.167761 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:53.167767 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:53.167974 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:53.168125 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:53.169264 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.170329 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.171527 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.172232 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.172333 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:53.176400 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:53.176411 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:53.176427 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:53.176461 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:53.179515 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.180229 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.180237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.180240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.180358 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.180887 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.180891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.180893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.181349 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:53.181594 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:53.181803 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 02:45:53.181995 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":284,"threads_started":1}
I20260929 02:45:53.183677 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:53.265898 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 635
I20260929 02:45:53.287164 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 769
Sep 29 02:45:53 dist-test-slave-6tjh krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:45:53.305635 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:53.288314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35488 (local address 127.0.97.3:42503)
0929 02:45:53.289002 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:53.289009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:53.289039 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:45:53.289097 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:53.289103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:53.289315 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:53.289462 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:53.291157 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.292212 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.293606 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:53.294226 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:53.294336 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:53.298852 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:53.298869 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:53.298885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:53.298927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:53.302369 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.303136 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.303144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.303146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.303294 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:53.303834 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:53.303840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:53.303842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:53.304313 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:53.304725 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:53.305014 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:45:53.305237 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":404,"threads_started":1}
I20260929 02:45:53.306910 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:53.391063 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 769
I20260929 02:45:53.407265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 408
I20260929 02:45:53.408520 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:53.509747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 408
2026-09-29T02:45:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3977 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:45:53 dist-test-slave-6tjh krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:45:53 dist-test-slave-6tjh krb5kdc[918](info): set up 2 sockets
Sep 29 02:45:53 dist-test-slave-6tjh krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260929 02:45:54.604415 388 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:45:54 dist-test-slave-6tjh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:45:54.784363 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44887
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:55.101763 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:55.102069 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:55.102232 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:55.109503 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:45:55.109617 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:55.109704 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:55.109805 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:45:55.109892 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:45:55.119805 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:55.121198 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:55.122582 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:55.134727 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:55.134701 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:55.134774 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:55.135584 931 server_base.cc:1034] running on GCE node
I20260929 02:45:55.136417 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:55.137943 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:55.139192 931 hybrid_clock.cc:648] HybridClock initialized: now 1790649955139129 us; error 49 us; skew 500 ppm
I20260929 02:45:55.139572 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:55 dist-test-slave-6tjh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:55.146688 931 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:45:55.148229 931 webserver.cc:460] Webserver started at http://127.0.97.62:35147/ using document root <none> and password file <none>
I20260929 02:45:55.148959 931 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:55.149060 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:55.149317 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:55.152315 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "6c2920f6722e4c9a8493bdc05f0c8782"
format_stamp: "Formatted at 2026-09-29 02:45:55 on dist-test-slave-6tjh"
server_key: "ebf26a110910b7b985c42c93e7cb9171"
server_key_iv: "c07746330aa724bf1c5526d4e2fca9b2"
server_key_version: "encryptionkey@0"
I20260929 02:45:55.153149 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "6c2920f6722e4c9a8493bdc05f0c8782"
format_stamp: "Formatted at 2026-09-29 02:45:55 on dist-test-slave-6tjh"
server_key: "ebf26a110910b7b985c42c93e7cb9171"
server_key_iv: "c07746330aa724bf1c5526d4e2fca9b2"
server_key_version: "encryptionkey@0"
I20260929 02:45:55.158478 931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:45:55.162051 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:55.163447 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:45:55.163595 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "6c2920f6722e4c9a8493bdc05f0c8782"
format_stamp: "Formatted at 2026-09-29 02:45:55 on dist-test-slave-6tjh"
server_key: "ebf26a110910b7b985c42c93e7cb9171"
server_key_iv: "c07746330aa724bf1c5526d4e2fca9b2"
server_key_version: "encryptionkey@0"
I20260929 02:45:55.163769 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:55.178464 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:55.182716 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:55.182942 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:55.201014 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44887
I20260929 02:45:55.201037 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44887 every 8 connection(s)
I20260929 02:45:55.202435 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:45:55.206095 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:45:55.211128 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 931
I20260929 02:45:55.211392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:45:55.212054 388 external_mini_cluster.cc:1428] Setting key c1d8403b233a9d93afee06b9cde1bb5b
I20260929 02:45:55.217027 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782: Bootstrap starting.
I20260929 02:45:55.220770 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782: Neither blocks nor log segments found. Creating new log.
I20260929 02:45:55.221995 998 log.cc:824] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782: Log is configured to *not* fsync() on all Append() calls
I20260929 02:45:55.225476 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782: No bootstrap required, opened a new log
Sep 29 02:45:55 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:55.235922 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44887 } }
I20260929 02:45:55.236411 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:45:55.236521 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2920f6722e4c9a8493bdc05f0c8782, State: Initialized, Role: FOLLOWER
I20260929 02:45:55.237064 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:55.214084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.0.97.62:44887)
0929 02:45:55.214949 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:55.214960 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:55.215004 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:45:55.216743 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:55.216766 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:55.217192 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:55.217543 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:55.219311 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:55.220609 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:55.222349 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:55.223466 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:55.223596 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:55.228207 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:55.228217 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:55.228237 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:55.228279 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:55.232407 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:55.233148 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:55.233161 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:55.233172 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:55.233382 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:55.233756 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:55.233761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:55.233765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:55.234504 (+ 739us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:55.234792 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:55.235195 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 02:45:55.235625 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":408,"threads_started":1}
I20260929 02:45:55.237244 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44887 } }
I20260929 02:45:55.237541 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:45:55.237748 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:45:55.237941 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:45:55.240242 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44887 } }
I20260929 02:45:55.240804 998 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2920f6722e4c9a8493bdc05f0c8782; no voters:
I20260929 02:45:55.241369 998 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:45:55.241503 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:45:55.242348 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2920f6722e4c9a8493bdc05f0c8782, State: Running, Role: LEADER
I20260929 02:45:55.242893 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44887 } }
I20260929 02:45:55.243513 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:45:55.245801 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44887 } } }
I20260929 02:45:55.245769 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2920f6722e4c9a8493bdc05f0c8782. Latest consensus state: current_term: 1 leader_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2920f6722e4c9a8493bdc05f0c8782" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44887 } } }
I20260929 02:45:55.246210 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [sys.catalog]: This master's current role is: LEADER
I20260929 02:45:55.246572 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782 [sys.catalog]: This master's current role is: LEADER
I20260929 02:45:55.247846 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:45:55.253396 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:45:55.261473 1008 catalog_manager.cc:1347] Generated new cluster ID: 0d738503af9d431b93cd9324eaaa78dd
I20260929 02:45:55.261605 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:45:55.298723 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:45:55.300060 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:45:55.306496 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2920f6722e4c9a8493bdc05f0c8782: Generated new TSK 0
I20260929 02:45:55.307324 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:45:55.356637 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44887
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:55.678375 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:55.678720 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:55.678895 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:55.686152 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:55.686321 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:55.686489 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:45:55.696233 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:55.697536 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:55.698801 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:55.711668 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:55.711746 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:55.711762 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:55.712733 1024 server_base.cc:1034] running on GCE node
I20260929 02:45:55.713394 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:55.714380 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:55.715659 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790649955715578 us; error 59 us; skew 500 ppm
I20260929 02:45:55.716050 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:55 dist-test-slave-6tjh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:55.719434 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:45:55.721011 1024 webserver.cc:460] Webserver started at http://127.0.97.1:39059/ using document root <none> and password file <none>
I20260929 02:45:55.721694 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:55.721798 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:55.722141 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:55.726351 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "425c70fd445f406db43de09c82cb7e34"
format_stamp: "Formatted at 2026-09-29 02:45:55 on dist-test-slave-6tjh"
server_key: "6421891d3c5517713f98a4443595ca5e"
server_key_iv: "fc451328a85941bf14d2320ce22c145c"
server_key_version: "encryptionkey@0"
I20260929 02:45:55.727567 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "425c70fd445f406db43de09c82cb7e34"
format_stamp: "Formatted at 2026-09-29 02:45:55 on dist-test-slave-6tjh"
server_key: "6421891d3c5517713f98a4443595ca5e"
server_key_iv: "fc451328a85941bf14d2320ce22c145c"
server_key_version: "encryptionkey@0"
I20260929 02:45:55.734429 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:45:55.738166 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:55.739600 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:45:55.739800 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "425c70fd445f406db43de09c82cb7e34"
format_stamp: "Formatted at 2026-09-29 02:45:55 on dist-test-slave-6tjh"
server_key: "6421891d3c5517713f98a4443595ca5e"
server_key_iv: "fc451328a85941bf14d2320ce22c145c"
server_key_version: "encryptionkey@0"
I20260929 02:45:55.740016 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:55.764490 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:55.768919 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:55.769243 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:55.770349 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:45:55.771669 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:45:55.771780 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:55.771931 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:45:55.772006 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:55.806476 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41931
I20260929 02:45:55.806495 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41931 every 8 connection(s)
I20260929 02:45:55.807838 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:45:55.815560 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 1024
I20260929 02:45:55.815802 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:45:55.816473 388 external_mini_cluster.cc:1428] Setting key 4e0ba337167f3d5b15b28e6e1fbfe074
Sep 29 02:45:55 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649955, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:55.830538 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:55.810722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47315 (local address 127.0.97.62:44887)
0929 02:45:55.811181 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:55.811185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:55.812379 (+ 1194us) server_negotiation.cc:407] Connection header received
0929 02:45:55.814499 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:55.814506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:55.814609 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:55.814753 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:55.816591 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:55.817534 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:55.819382 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:55.819831 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:55.819983 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:55.824002 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:55.824008 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:55.824011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:55.824043 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:55.826723 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:55.827848 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:55.827852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:55.827855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:55.827982 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:55.828604 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:55.828608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:55.828611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:55.828900 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:55.829015 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:55.830107 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0929 02:45:55.830288 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260929 02:45:55.832417 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:55.811384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44887 (local address 127.0.97.1:47315)
0929 02:45:55.812185 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:55.812227 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:55.814197 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:55.815012 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:55.815021 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:55.815627 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:45:55.816372 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:55.816391 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:55.817737 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:55.817742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:55.819223 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:55.819236 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:55.820078 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:55.820089 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:55.820360 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:55.821188 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:45:55.821213 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:45:55.823727 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:45:55.826899 (+ 3172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:55.826913 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:55.826937 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:55.827702 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:55.828110 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:55.828119 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:55.828127 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:55.828415 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:55.829042 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:45:55.829055 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:45:55.829622 (+ 567us) client_negotiation.cc:769] Sending connection context
0929 02:45:55.830013 (+ 391us) client_negotiation.cc:240] Negotiation successful
0929 02:45:55.830711 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":316,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:45:55.834344 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44887
I20260929 02:45:55.834750 1152 heartbeater.cc:464] Registering TS with master...
I20260929 02:45:55.835484 1152 heartbeater.cc:511] Master 127.0.97.62:44887 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:45:55.837815 963 ts_manager.cc:194] Registered new tserver with Master: 425c70fd445f406db43de09c82cb7e34 (127.0.97.1:41931)
I20260929 02:45:55.839273 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:45:55.860816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44887
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:56.191416 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:56.191706 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:56.191880 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:56.199018 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:56.199123 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:56.199318 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:45:56.209192 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:56.210423 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:56.211706 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:56.224771 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:56.224841 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:56.224854 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:56.225121 1158 server_base.cc:1034] running on GCE node
I20260929 02:45:56.225997 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:56.226846 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:56.228046 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790649956228004 us; error 30 us; skew 500 ppm
I20260929 02:45:56.228376 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:56 dist-test-slave-6tjh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:56.231614 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:45:56.233184 1158 webserver.cc:460] Webserver started at http://127.0.97.2:37585/ using document root <none> and password file <none>
I20260929 02:45:56.233877 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:56.234006 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:56.234321 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:56.237223 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "f4ecb35c7bb74fc691c12de9e37360da"
format_stamp: "Formatted at 2026-09-29 02:45:56 on dist-test-slave-6tjh"
server_key: "80b8cf2f4f49b574a7a846f74af81b40"
server_key_iv: "c11b5eb6b3505e4016364590c7732733"
server_key_version: "encryptionkey@0"
I20260929 02:45:56.238103 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "f4ecb35c7bb74fc691c12de9e37360da"
format_stamp: "Formatted at 2026-09-29 02:45:56 on dist-test-slave-6tjh"
server_key: "80b8cf2f4f49b574a7a846f74af81b40"
server_key_iv: "c11b5eb6b3505e4016364590c7732733"
server_key_version: "encryptionkey@0"
I20260929 02:45:56.243247 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:45:56.246752 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:56.248061 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:45:56.248229 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "f4ecb35c7bb74fc691c12de9e37360da"
format_stamp: "Formatted at 2026-09-29 02:45:56 on dist-test-slave-6tjh"
server_key: "80b8cf2f4f49b574a7a846f74af81b40"
server_key_iv: "c11b5eb6b3505e4016364590c7732733"
server_key_version: "encryptionkey@0"
I20260929 02:45:56.248395 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:56.265528 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:56.270105 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:56.270397 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:56.271451 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:45:56.272835 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:45:56.272913 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:56.273061 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:45:56.273110 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:56.306252 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43763
I20260929 02:45:56.306293 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43763 every 8 connection(s)
I20260929 02:45:56.307585 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:45:56.309140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 1158
I20260929 02:45:56.309373 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:45:56.309880 388 external_mini_cluster.cc:1428] Setting key aa92e50565639f5e8d826cdd60d2316a
Sep 29 02:45:56 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 02:45:56.331352 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.310425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50273 (local address 127.0.97.62:44887)
0929 02:45:56.310624 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:56.310630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:56.312007 (+ 1377us) server_negotiation.cc:407] Connection header received
0929 02:45:56.314926 (+ 2919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:56.314929 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:56.314991 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:56.315080 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:56.317278 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.318465 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.320000 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.320822 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.320972 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.325361 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:56.325366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:56.325369 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:56.325401 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:56.328031 (+ 2630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:56.328882 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:56.328886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:56.328888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:56.329014 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:56.329479 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:56.329483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:56.329486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:56.329760 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:56.329878 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:56.330819 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 02:45:56.331007 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:45:56.332861 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.311008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44887 (local address 127.0.97.2:50273)
0929 02:45:56.311852 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:56.311895 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:56.314614 (+ 2719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:56.315371 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:56.315386 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:56.316214 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:45:56.317095 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.317116 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.318624 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.318627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.319859 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.319870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.321026 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.321031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.321224 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.321938 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:45:56.321977 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:45:56.325118 (+ 3141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:45:56.328173 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:56.328182 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:56.328206 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:56.328725 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:56.329116 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:56.329119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:56.329122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:56.329297 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:56.329879 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:45:56.329885 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:45:56.330322 (+ 437us) client_negotiation.cc:769] Sending connection context
0929 02:45:56.330620 (+ 298us) client_negotiation.cc:240] Negotiation successful
0929 02:45:56.331190 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":333,"threads_started":1}
I20260929 02:45:56.334685 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44887
I20260929 02:45:56.335029 1286 heartbeater.cc:464] Registering TS with master...
I20260929 02:45:56.335759 1286 heartbeater.cc:511] Master 127.0.97.62:44887 requested a full tablet report, sending...
I20260929 02:45:56.337320 963 ts_manager.cc:194] Registered new tserver with Master: f4ecb35c7bb74fc691c12de9e37360da (127.0.97.2:43763)
I20260929 02:45:56.338152 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50273
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:45:56.354290 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44887
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:56.675736 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:56.676028 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:56.676121 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:56.683260 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:56.683377 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:56.683490 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:45:56.693269 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:44887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:56.694514 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:56.695792 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:56.709156 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:56.709400 1291 server_base.cc:1034] running on GCE node
W20260929 02:45:56.709203 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:56.709224 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:56.710536 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:56.711447 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:56.712664 1291 hybrid_clock.cc:648] HybridClock initialized: now 1790649956712589 us; error 50 us; skew 500 ppm
I20260929 02:45:56.712960 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:56 dist-test-slave-6tjh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:56.716164 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:45:56.717707 1291 webserver.cc:460] Webserver started at http://127.0.97.3:37483/ using document root <none> and password file <none>
I20260929 02:45:56.718402 1291 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:56.718530 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:56.718845 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:56.721733 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "85a11f764a72473d9072fb26f53545b6"
format_stamp: "Formatted at 2026-09-29 02:45:56 on dist-test-slave-6tjh"
server_key: "6f5eaf152ab80115d12dbf09557e4af5"
server_key_iv: "9f2895322f60ad3ad3f4127f98c7de3f"
server_key_version: "encryptionkey@0"
I20260929 02:45:56.722532 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "85a11f764a72473d9072fb26f53545b6"
format_stamp: "Formatted at 2026-09-29 02:45:56 on dist-test-slave-6tjh"
server_key: "6f5eaf152ab80115d12dbf09557e4af5"
server_key_iv: "9f2895322f60ad3ad3f4127f98c7de3f"
server_key_version: "encryptionkey@0"
I20260929 02:45:56.727893 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 02:45:56.731820 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:56.733330 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:45:56.733502 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "85a11f764a72473d9072fb26f53545b6"
format_stamp: "Formatted at 2026-09-29 02:45:56 on dist-test-slave-6tjh"
server_key: "6f5eaf152ab80115d12dbf09557e4af5"
server_key_iv: "9f2895322f60ad3ad3f4127f98c7de3f"
server_key_version: "encryptionkey@0"
I20260929 02:45:56.733700 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:56.749143 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:56.753849 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:56.754135 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:56.755157 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:45:56.756733 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:45:56.756839 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:56.756984 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:45:56.757059 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:56.789923 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38483
I20260929 02:45:56.789966 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38483 every 8 connection(s)
I20260929 02:45:56.791226 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:45:56.801611 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 1291
I20260929 02:45:56.801851 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:45:56.802281 388 external_mini_cluster.cc:1428] Setting key 4574853f00922b3ffb0795237f5460df
Sep 29 02:45:56 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:56.811254 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.793753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34493 (local address 127.0.97.62:44887)
0929 02:45:56.793935 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:56.793939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:56.795100 (+ 1161us) server_negotiation.cc:407] Connection header received
0929 02:45:56.796943 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:56.796949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:56.797039 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:56.797158 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:56.798603 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.799372 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.801169 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.801581 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.801717 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.805741 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:56.805745 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:56.805749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:56.805779 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:56.808227 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:56.808994 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:56.809003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:56.809006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:56.809168 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:56.809599 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:56.809604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:56.809607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:56.809928 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:56.810067 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:56.810714 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 02:45:56.810925 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:45:56.812111 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.794230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44887 (local address 127.0.97.3:34493)
0929 02:45:56.794938 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:56.794978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:56.796678 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:56.797381 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:56.797389 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:56.797903 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:45:56.798433 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.798450 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.799529 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.799536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.800999 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.801011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.801811 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.801819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.802496 (+ 677us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.803178 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:45:56.803200 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:45:56.805544 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:45:56.808382 (+ 2838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:56.808391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:56.808406 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:56.808871 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:56.809286 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:56.809292 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:56.809296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:56.809485 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:56.810053 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:45:56.810060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:45:56.810366 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 02:45:56.810604 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 02:45:56.811091 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":266,"threads_started":1}
I20260929 02:45:56.813473 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.62:44887
I20260929 02:45:56.813730 1419 heartbeater.cc:464] Registering TS with master...
I20260929 02:45:56.814380 1419 heartbeater.cc:511] Master 127.0.97.62:44887 requested a full tablet report, sending...
I20260929 02:45:56.815689 963 ts_manager.cc:194] Registered new tserver with Master: 85a11f764a72473d9072fb26f53545b6 (127.0.97.3:38483)
I20260929 02:45:56.816604 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34493
I20260929 02:45:56.817595 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:45:56.836759 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.824278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.0.97.62:44887)
0929 02:45:56.824479 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:56.824489 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:56.825029 (+ 540us) server_negotiation.cc:407] Connection header received
0929 02:45:56.825168 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:56.825172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:56.825244 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:56.825330 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:56.826704 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.827397 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.828691 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.829088 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.829182 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.831561 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:56.831566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:56.831569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:56.831593 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:56.834038 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:56.834727 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:56.834732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:56.834734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:56.834886 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:56.835236 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:56.835240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:56.835242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:56.835624 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:56.835831 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:56.836106 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 02:45:56.836310 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:45:56.843083 1152 heartbeater.cc:503] Master 127.0.97.62:44887 was elected leader, sending a full tablet report...
I20260929 02:45:56.843905 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:45:56.851897 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:45:56.889081 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.863962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.97.3:38483)
0929 02:45:56.880237 (+ 16275us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:56.880248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:56.880297 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:45:56.880467 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:56.880479 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:56.880782 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:56.880999 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:45:56.881804 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.883258 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.886414 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.887869 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.888024 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.888245 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:56.888521 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 02:45:56.888718 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16102,"thread_start_us":15958,"threads_started":1}
I20260929 02:45:56.889416 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.864806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.0.97.2:43763)
0929 02:45:56.872176 (+ 7370us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:56.872188 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:56.880353 (+ 8165us) server_negotiation.cc:407] Connection header received
0929 02:45:56.880653 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:56.880660 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:56.880851 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:56.880999 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:45:56.882353 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.884068 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.886346 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.887820 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.887925 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.888137 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:56.888946 (+ 809us) server_negotiation.cc:299] Negotiation successful
0929 02:45:56.889103 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7169,"thread_start_us":7052,"threads_started":1}
I20260929 02:45:56.889469 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.863733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41931 (local address 127.0.0.1:41256)
0929 02:45:56.878951 (+ 15218us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:56.878969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:56.879074 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:56.881280 (+ 2206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:56.881284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:56.881319 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:45:56.882163 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.882179 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.884259 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.884266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.886636 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.886648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.888656 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.888661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.888807 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.888839 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:45:56.888929 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 02:45:56.889045 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15077,"mutex_wait_us":100,"spinlock_wait_cycles":1920,"thread_start_us":15409,"threads_started":1}
I20260929 02:45:56.889529 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.863665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38483 (local address 127.0.0.1:41946)
0929 02:45:56.877269 (+ 13604us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:56.877337 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:56.877593 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:56.881032 (+ 3439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:56.881040 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:56.881081 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:45:56.881589 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.881603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.883461 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.883468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.886220 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.886235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.888071 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.888077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.888291 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.888337 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 02:45:56.888475 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:45:56.888710 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13229,"mutex_wait_us":66,"thread_start_us":16302,"threads_started":1}
I20260929 02:45:56.890182 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.864204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43763 (local address 127.0.0.1:59736)
0929 02:45:56.880231 (+ 16027us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:56.880249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:56.880451 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:56.881269 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:56.881274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:56.881311 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:45:56.882125 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.882148 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.884251 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.884255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.886167 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:56.886178 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.888328 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.888334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:56.888488 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.888523 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:45:56.888641 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 02:45:56.888771 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15886,"mutex_wait_us":53,"spinlock_wait_cycles":1920,"thread_start_us":13654,"threads_started":1}
I20260929 02:45:56.890520 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:56.864751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.0.97.1:41931)
0929 02:45:56.880698 (+ 15947us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:56.880705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:56.880728 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:45:56.880800 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:56.880805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:56.881031 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:56.881321 (+ 290us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:45:56.882373 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.884068 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.886794 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:56.888165 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:56.888290 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:56.888484 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:56.890084 (+ 1600us) server_negotiation.cc:299] Negotiation successful
0929 02:45:56.890237 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15771,"thread_start_us":15628,"threads_started":1}
I20260929 02:45:56.893483 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 0abe8832384d409a8af40e62acd52ba3 (DEFAULT_TABLE table=test-table [id=90f27405eb354719808f96d725fe52fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:45:56.893505 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 0abe8832384d409a8af40e62acd52ba3 (DEFAULT_TABLE table=test-table [id=90f27405eb354719808f96d725fe52fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:45:56.894685 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abe8832384d409a8af40e62acd52ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:45:56.894879 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abe8832384d409a8af40e62acd52ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:45:56.894590 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 0abe8832384d409a8af40e62acd52ba3 (DEFAULT_TABLE table=test-table [id=90f27405eb354719808f96d725fe52fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:45:56.895787 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0abe8832384d409a8af40e62acd52ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:45:56.905915 1437 tablet_bootstrap.cc:492] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da: Bootstrap starting.
I20260929 02:45:56.909793 1438 tablet_bootstrap.cc:492] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6: Bootstrap starting.
I20260929 02:45:56.910845 1437 tablet_bootstrap.cc:654] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da: Neither blocks nor log segments found. Creating new log.
I20260929 02:45:56.912210 1437 log.cc:824] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da: Log is configured to *not* fsync() on all Append() calls
I20260929 02:45:56.913249 1439 tablet_bootstrap.cc:492] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34: Bootstrap starting.
I20260929 02:45:56.913560 1438 tablet_bootstrap.cc:654] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6: Neither blocks nor log segments found. Creating new log.
I20260929 02:45:56.914990 1438 log.cc:824] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:45:56.917392 1439 tablet_bootstrap.cc:654] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34: Neither blocks nor log segments found. Creating new log.
I20260929 02:45:56.917800 1437 tablet_bootstrap.cc:492] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da: No bootstrap required, opened a new log
I20260929 02:45:56.918002 1438 tablet_bootstrap.cc:492] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6: No bootstrap required, opened a new log
I20260929 02:45:56.918253 1438 ts_tablet_manager.cc:1397] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 02:45:56.918504 1437 ts_tablet_manager.cc:1397] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 02:45:56.918546 1439 log.cc:824] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34: Log is configured to *not* fsync() on all Append() calls
I20260929 02:45:56.921175 1439 tablet_bootstrap.cc:492] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34: No bootstrap required, opened a new log
I20260929 02:45:56.921428 1439 ts_tablet_manager.cc:1397] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260929 02:45:56.928987 1439 raft_consensus.cc:348] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:56.929548 1439 raft_consensus.cc:374] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:45:56.929649 1439 raft_consensus.cc:729] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425c70fd445f406db43de09c82cb7e34, State: Initialized, Role: FOLLOWER
I20260929 02:45:56.929716 1438 raft_consensus.cc:348] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:56.929744 1437 raft_consensus.cc:348] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:56.930160 1437 raft_consensus.cc:374] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:45:56.930160 1438 raft_consensus.cc:374] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:45:56.930269 1438 raft_consensus.cc:729] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a11f764a72473d9072fb26f53545b6, State: Initialized, Role: FOLLOWER
I20260929 02:45:56.930262 1437 raft_consensus.cc:729] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ecb35c7bb74fc691c12de9e37360da, State: Initialized, Role: FOLLOWER
I20260929 02:45:56.930384 1439 consensus_queue.cc:260] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:56.930891 1437 consensus_queue.cc:260] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:56.930891 1438 consensus_queue.cc:260] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:56.932194 1286 heartbeater.cc:503] Master 127.0.97.62:44887 was elected leader, sending a full tablet report...
I20260929 02:45:56.933125 1437 ts_tablet_manager.cc:1428] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 02:45:56.934077 1419 heartbeater.cc:503] Master 127.0.97.62:44887 was elected leader, sending a full tablet report...
I20260929 02:45:56.934996 1438 ts_tablet_manager.cc:1428] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260929 02:45:56.938053 1439 ts_tablet_manager.cc:1428] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34: Time spent starting tablet: real 0.016s user 0.005s sys 0.011s
W20260929 02:45:57.042928 1420 tablet.cc:2369] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:45:57.059556 1287 tablet.cc:2369] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:45:57.060140 1153 tablet.cc:2369] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:45:57.074167 1445 raft_consensus.cc:482] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:45:57.074559 1445 raft_consensus.cc:504] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:57.076623 1445 leader_election.cc:288] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 425c70fd445f406db43de09c82cb7e34 (127.0.97.1:41931), f4ecb35c7bb74fc691c12de9e37360da (127.0.97.2:43763)
I20260929 02:45:57.082888 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.076817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41931 (local address 127.0.97.3:53987)
0929 02:45:57.077105 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:57.077131 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:57.077296 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:57.077788 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:57.077797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:57.077833 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:45:57.078416 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:57.078431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.079484 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.079492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:57.081146 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:57.081160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.082345 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.082349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:57.082478 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.082502 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:45:57.082575 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 02:45:57.082679 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":1}
I20260929 02:45:57.083065 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.077075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53987 (local address 127.0.97.1:41931)
0929 02:45:57.077302 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.077310 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.077336 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:45:57.077458 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.077466 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.077569 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.077683 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:45:57.078637 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.079351 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.081318 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.082185 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.082329 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.082435 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.082686 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.082828 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:45:57.084460 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.077411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43735 (local address 127.0.97.2:43763)
0929 02:45:57.077687 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.077695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.078553 (+ 858us) server_negotiation.cc:407] Connection header received
0929 02:45:57.078848 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.078853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.078913 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.079012 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:45:57.079551 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.080260 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.082245 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.083501 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.083673 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.083758 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.084172 (+ 414us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.084273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260929 02:45:57.084239 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abe8832384d409a8af40e62acd52ba3" candidate_uuid: "85a11f764a72473d9072fb26f53545b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "425c70fd445f406db43de09c82cb7e34" is_pre_election: true
I20260929 02:45:57.084681 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.077117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43763 (local address 127.0.97.3:43735)
0929 02:45:57.078430 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:57.078451 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:57.078587 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:57.079028 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:57.079032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:57.079060 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:45:57.079371 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:57.079379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.080411 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.080417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:57.082062 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:57.082074 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.083787 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.083794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:57.083974 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.084004 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:45:57.084114 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:45:57.084280 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1170,"mutex_wait_us":24,"thread_start_us":472,"threads_started":1}
I20260929 02:45:57.084894 1107 raft_consensus.cc:2454] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a11f764a72473d9072fb26f53545b6 in term 0.
I20260929 02:45:57.085613 1309 leader_election.cc:302] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 425c70fd445f406db43de09c82cb7e34, 85a11f764a72473d9072fb26f53545b6; no voters:
I20260929 02:45:57.085739 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abe8832384d409a8af40e62acd52ba3" candidate_uuid: "85a11f764a72473d9072fb26f53545b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4ecb35c7bb74fc691c12de9e37360da" is_pre_election: true
I20260929 02:45:57.085997 1445 raft_consensus.cc:2781] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:45:57.086162 1445 raft_consensus.cc:482] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:45:57.086261 1445 raft_consensus.cc:3037] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:45:57.086329 1241 raft_consensus.cc:2454] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85a11f764a72473d9072fb26f53545b6 in term 0.
I20260929 02:45:57.088624 1445 raft_consensus.cc:504] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:57.089321 1445 leader_election.cc:288] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [CANDIDATE]: Term 1 election: Requested vote from peers 425c70fd445f406db43de09c82cb7e34 (127.0.97.1:41931), f4ecb35c7bb74fc691c12de9e37360da (127.0.97.2:43763)
I20260929 02:45:57.089804 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abe8832384d409a8af40e62acd52ba3" candidate_uuid: "85a11f764a72473d9072fb26f53545b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "425c70fd445f406db43de09c82cb7e34"
I20260929 02:45:57.089854 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0abe8832384d409a8af40e62acd52ba3" candidate_uuid: "85a11f764a72473d9072fb26f53545b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4ecb35c7bb74fc691c12de9e37360da"
I20260929 02:45:57.090004 1107 raft_consensus.cc:3037] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:45:57.090065 1241 raft_consensus.cc:3037] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:45:57.092212 1107 raft_consensus.cc:2454] T 0abe8832384d409a8af40e62acd52ba3 P 425c70fd445f406db43de09c82cb7e34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a11f764a72473d9072fb26f53545b6 in term 1.
I20260929 02:45:57.092695 1309 leader_election.cc:302] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 425c70fd445f406db43de09c82cb7e34, 85a11f764a72473d9072fb26f53545b6; no voters:
I20260929 02:45:57.092727 1241 raft_consensus.cc:2454] T 0abe8832384d409a8af40e62acd52ba3 P f4ecb35c7bb74fc691c12de9e37360da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85a11f764a72473d9072fb26f53545b6 in term 1.
I20260929 02:45:57.092980 1445 raft_consensus.cc:2781] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:45:57.093449 1445 raft_consensus.cc:686] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [term 1 LEADER]: Becoming Leader. State: Replica: 85a11f764a72473d9072fb26f53545b6, State: Running, Role: LEADER
I20260929 02:45:57.093936 1445 consensus_queue.cc:237] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } }
I20260929 02:45:57.098838 961 catalog_manager.cc:5495] T 0abe8832384d409a8af40e62acd52ba3 P 85a11f764a72473d9072fb26f53545b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85a11f764a72473d9072fb26f53545b6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "85a11f764a72473d9072fb26f53545b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85a11f764a72473d9072fb26f53545b6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "425c70fd445f406db43de09c82cb7e34" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4ecb35c7bb74fc691c12de9e37360da" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43763 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649954, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:45:57.135200 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.118198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41258 (local address 127.0.97.1:41931)
0929 02:45:57.118378 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.118382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.118936 (+ 554us) server_negotiation.cc:407] Connection header received
0929 02:45:57.119155 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.119168 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.119258 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.119385 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:57.121559 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.122359 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.123738 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.124205 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.124360 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.128973 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:57.128999 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:57.129015 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:57.129054 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:57.132223 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.133047 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.133054 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.133057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.133213 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.133669 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.133672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.133674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.134036 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:57.134270 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.134743 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.134918 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649957, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:45:57.182562 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.168331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41270 (local address 127.0.97.1:41931)
0929 02:45:57.168579 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.168588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.168961 (+ 373us) server_negotiation.cc:407] Connection header received
0929 02:45:57.169205 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.169210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.169278 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.169404 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:57.171105 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.171842 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.173169 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.173583 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.173726 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.177527 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:57.177532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:57.177535 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:57.177568 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:57.179985 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.180660 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.180665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.180669 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.180817 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.181200 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.181204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.181206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.181477 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:57.181739 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.182035 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.182264 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:45:57.226825 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.212840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.0.97.1:41931)
0929 02:45:57.213064 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.213070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.213264 (+ 194us) server_negotiation.cc:407] Connection header received
0929 02:45:57.213476 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.213487 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.213551 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.213658 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:57.215112 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.215812 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.217114 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.217539 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.217689 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.221620 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:57.221625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:57.221628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:57.221664 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:57.224108 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.224868 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.224873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.224875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.225001 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.225397 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.225406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.225408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.225692 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:57.225970 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.226250 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.226487 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260929 02:45:57.228633 1067 tablet_service.cc:2908] Rejecting scan request for tablet 0abe8832384d409a8af40e62acd52ba3: Uninitialized: clean time has not yet been initialized
I20260929 02:45:57.232009 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1024
I20260929 02:45:57.245680 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.233144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.0.97.1:41931)
0929 02:45:57.233321 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.233326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.233741 (+ 415us) server_negotiation.cc:407] Connection header received
0929 02:45:57.233863 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.233867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.233945 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.234033 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:57.235204 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.235917 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.237193 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.237686 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.237826 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.240092 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:57.240097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:57.240100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:57.240144 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:57.243001 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.243836 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.243841 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.243843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.243963 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.244318 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.244322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.244355 (+ 33us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.244782 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:57.245006 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.245232 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.245395 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:45:57.246332 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:57.336735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 1024
I20260929 02:45:57.354976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1158
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:45:57.370898 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.355715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.0.97.2:43763)
0929 02:45:57.355873 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.355878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.355914 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:45:57.356086 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.356090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.356190 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.356287 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:57.357861 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.358667 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.360047 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.360481 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.360723 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.365009 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:57.365025 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:57.365041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:57.365081 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:57.368300 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.369075 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.369082 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.369084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.369215 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.369562 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.369565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.369567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.369941 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:57.370162 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.370466 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.370628 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:45:57.371521 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:57.462019 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 1158
I20260929 02:45:57.478384 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1291
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:45:57.493853 1462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:57.479108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.0.97.3:38483)
0929 02:45:57.479569 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:57.479574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:57.479591 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:45:57.479650 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:57.479655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:57.479719 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:57.479830 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:57.481329 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.482024 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.483302 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:57.483782 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:57.483898 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:57.488093 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:57.488108 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:57.488124 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:57.488164 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:57.491144 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.491922 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.491930 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.491932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.492073 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:57.492409 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:57.492412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:57.492414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:57.492893 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:57.493114 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:57.493358 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 02:45:57.493521 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
I20260929 02:45:57.494494 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:57.583024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 1291
I20260929 02:45:57.598978 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 931
I20260929 02:45:57.600071 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:45:57.706347 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 931
2026-09-29T02:45:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4197 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[1467](info): set up 2 sockets
Sep 29 02:45:57 dist-test-slave-6tjh krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260929 02:45:58.796765 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:45:58 dist-test-slave-6tjh krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:45:58.980074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39735
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:59.301847 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:59.302158 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:59.302330 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:59.309597 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:45:59.309705 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:45:59.309785 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:59.309903 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:59.309988 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:45:59.310102 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:45:59.319980 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39735
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:59.321341 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:59.322705 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:59.337018 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:59.337069 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:59.337041 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:59.337857 1480 server_base.cc:1034] running on GCE node
I20260929 02:45:59.338687 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:59.340190 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:59.341446 1480 hybrid_clock.cc:648] HybridClock initialized: now 1790649959341369 us; error 57 us; skew 500 ppm
I20260929 02:45:59.341814 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:59 dist-test-slave-6tjh krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:59.344996 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:45:59.346441 1480 webserver.cc:460] Webserver started at http://127.0.97.62:45179/ using document root <none> and password file <none>
I20260929 02:45:59.347189 1480 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:59.347321 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:59.347656 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:59.350678 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "65435bd814ba47bdb3b921bfa615a1c1"
format_stamp: "Formatted at 2026-09-29 02:45:59 on dist-test-slave-6tjh"
server_key: "440ab62de1a1e7673328f7f1dda7c836"
server_key_iv: "0de6927ccc50e44a21ef80a23c5f81f1"
server_key_version: "encryptionkey@0"
I20260929 02:45:59.351497 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "65435bd814ba47bdb3b921bfa615a1c1"
format_stamp: "Formatted at 2026-09-29 02:45:59 on dist-test-slave-6tjh"
server_key: "440ab62de1a1e7673328f7f1dda7c836"
server_key_iv: "0de6927ccc50e44a21ef80a23c5f81f1"
server_key_version: "encryptionkey@0"
I20260929 02:45:59.357017 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:45:59.361025 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:59.362529 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:45:59.362773 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "65435bd814ba47bdb3b921bfa615a1c1"
format_stamp: "Formatted at 2026-09-29 02:45:59 on dist-test-slave-6tjh"
server_key: "440ab62de1a1e7673328f7f1dda7c836"
server_key_iv: "0de6927ccc50e44a21ef80a23c5f81f1"
server_key_version: "encryptionkey@0"
I20260929 02:45:59.363013 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:59.377933 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:59.382256 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:59.382530 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:59.403586 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39735
I20260929 02:45:59.403597 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39735 every 8 connection(s)
I20260929 02:45:59.405006 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:45:59.408061 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 1480
I20260929 02:45:59.408290 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:45:59.408923 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:45:59.409046 388 external_mini_cluster.cc:1428] Setting key 6e209c07cb8bcd4d1902dddbf78de21c
I20260929 02:45:59.423102 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1: Bootstrap starting.
Sep 29 02:45:59 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649958, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:45:59.426955 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1: Neither blocks nor log segments found. Creating new log.
I20260929 02:45:59.428331 1557 log.cc:824] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:45:59.431933 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1: No bootstrap required, opened a new log
I20260929 02:45:59.433475 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:59.411027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.0.97.62:39735)
0929 02:45:59.411835 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:59.411850 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:59.411908 (+ 58us) server_negotiation.cc:407] Connection header received
0929 02:45:59.413814 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:59.413842 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:59.414347 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:59.414816 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:59.416058 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:59.418009 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:59.419287 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:59.420376 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:59.420588 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:59.424720 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:59.424732 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:59.424753 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:59.424811 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:59.428767 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:59.429707 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:59.429714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:59.429721 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:59.429970 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:59.430337 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:59.430341 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:59.430344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:59.431019 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:59.431301 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 02:45:59.431649 (+ 348us) server_negotiation.cc:299] Negotiation successful
0929 02:45:59.432081 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":330,"threads_started":1}
I20260929 02:45:59.440318 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65435bd814ba47bdb3b921bfa615a1c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39735 } }
I20260929 02:45:59.440723 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:45:59.440836 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65435bd814ba47bdb3b921bfa615a1c1, State: Initialized, Role: FOLLOWER
I20260929 02:45:59.441365 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65435bd814ba47bdb3b921bfa615a1c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39735 } }
I20260929 02:45:59.441594 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:45:59.441742 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:45:59.441917 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:45:59.444226 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65435bd814ba47bdb3b921bfa615a1c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39735 } }
I20260929 02:45:59.444950 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65435bd814ba47bdb3b921bfa615a1c1; no voters:
I20260929 02:45:59.445542 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:45:59.445698 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:45:59.446436 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [term 1 LEADER]: Becoming Leader. State: Replica: 65435bd814ba47bdb3b921bfa615a1c1, State: Running, Role: LEADER
I20260929 02:45:59.446979 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65435bd814ba47bdb3b921bfa615a1c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39735 } }
I20260929 02:45:59.447695 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:45:59.450052 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65435bd814ba47bdb3b921bfa615a1c1. Latest consensus state: current_term: 1 leader_uuid: "65435bd814ba47bdb3b921bfa615a1c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65435bd814ba47bdb3b921bfa615a1c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39735 } } }
I20260929 02:45:59.450417 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:45:59.452951 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65435bd814ba47bdb3b921bfa615a1c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65435bd814ba47bdb3b921bfa615a1c1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39735 } } }
I20260929 02:45:59.453299 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1 [sys.catalog]: This master's current role is: LEADER
I20260929 02:45:59.453509 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:45:59.458853 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:45:59.467602 1568 catalog_manager.cc:1347] Generated new cluster ID: f31d00b870ce4ebbbbbe216b43081d60
I20260929 02:45:59.467728 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:45:59.495072 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:45:59.497182 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:45:59.510828 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1: Generated new TSK 0
I20260929 02:45:59.511749 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:45:59.564679 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39735
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:45:59.892850 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:45:59.893151 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:45:59.893255 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:45:59.900760 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:45:59.900889 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:45:59.900981 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:45:59.901078 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:45:59.911118 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:45:59.912405 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:45:59.913857 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:45:59.927011 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:59.927052 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:45:59.927091 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:45:59.928299 1583 server_base.cc:1034] running on GCE node
I20260929 02:45:59.929068 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:45:59.930029 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:45:59.931239 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790649959931179 us; error 38 us; skew 500 ppm
I20260929 02:45:59.931533 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:45:59 dist-test-slave-6tjh krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:45:59.934609 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:45:59.936084 1583 webserver.cc:460] Webserver started at http://127.0.97.1:45851/ using document root <none> and password file <none>
I20260929 02:45:59.936859 1583 fs_manager.cc:359] Metadata directory not provided
I20260929 02:45:59.936950 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:45:59.937296 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:45:59.940069 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "856cc9e75d03495eb5355b04b6209630"
format_stamp: "Formatted at 2026-09-29 02:45:59 on dist-test-slave-6tjh"
server_key: "fc90b17f5b0da5cc32ebe6a2d0a1d512"
server_key_iv: "510128107b79b751294fd006fb789263"
server_key_version: "encryptionkey@0"
I20260929 02:45:59.940999 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "856cc9e75d03495eb5355b04b6209630"
format_stamp: "Formatted at 2026-09-29 02:45:59 on dist-test-slave-6tjh"
server_key: "fc90b17f5b0da5cc32ebe6a2d0a1d512"
server_key_iv: "510128107b79b751294fd006fb789263"
server_key_version: "encryptionkey@0"
I20260929 02:45:59.946004 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:45:59.949656 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:59.951104 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:45:59.951274 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "856cc9e75d03495eb5355b04b6209630"
format_stamp: "Formatted at 2026-09-29 02:45:59 on dist-test-slave-6tjh"
server_key: "fc90b17f5b0da5cc32ebe6a2d0a1d512"
server_key_iv: "510128107b79b751294fd006fb789263"
server_key_version: "encryptionkey@0"
I20260929 02:45:59.951504 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:45:59.968788 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:45:59.973251 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:45:59.973532 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:45:59.976289 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:45:59.976389 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:45:59.976506 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:45:59.976601 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 02:45:59 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:00.000747 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:59.976976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48371 (local address 127.0.97.62:39735)
0929 02:45:59.977529 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0929 02:45:59.977534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:45:59.978593 (+ 1059us) server_negotiation.cc:407] Connection header received
0929 02:45:59.980743 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:45:59.980749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:45:59.980845 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:45:59.980961 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:45:59.983223 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:59.984233 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:59.986767 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:45:59.987180 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:45:59.987311 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:59.993390 (+ 6079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:45:59.993400 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:45:59.993408 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:45:59.993468 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:45:59.996904 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:59.998151 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:59.998160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:59.998166 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:59.998312 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:45:59.998819 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:45:59.998824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:45:59.998828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:45:59.999136 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:45:59.999291 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:00.000036 (+ 745us) server_negotiation.cc:299] Negotiation successful
0929 02:46:00.000257 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":338,"threads_started":1}
I20260929 02:46:00.002045 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:45:59.977608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39735 (local address 127.0.97.1:48371)
0929 02:45:59.978403 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0929 02:45:59.978448 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:45:59.980385 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:45:59.981241 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:45:59.981255 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:45:59.982084 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:45:59.983025 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:59.983048 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:59.984471 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:59.984485 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:59.986613 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:45:59.986624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:45:59.987401 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:45:59.987409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:45:59.987740 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:45:59.988650 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:45:59.988693 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:45:59.993018 (+ 4325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:45:59.997136 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:59.997146 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:59.997170 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:59.997974 (+ 804us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:59.998442 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:45:59.998448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:45:59.998453 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:45:59.998684 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:45:59.999279 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:45:59.999288 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:45:59.999652 (+ 364us) client_negotiation.cc:769] Sending connection context
0929 02:45:59.999945 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 02:46:00.000489 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":347,"threads_started":1}
I20260929 02:46:00.029027 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:00.011950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47637 (local address 127.0.97.62:39735)
0929 02:46:00.012173 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:00.012182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:00.013590 (+ 1408us) server_negotiation.cc:407] Connection header received
0929 02:46:00.013925 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:00.013935 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:00.014050 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:00.014179 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:00.015820 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.016608 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.019229 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.019623 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.019731 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:00.022622 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:00.022627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:00.022631 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:00.022663 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:00.025787 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:00.026755 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:00.026762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:00.026766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:00.026931 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:00.027415 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:00.027424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:00.027428 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:00.027729 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:00.027892 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:00.028352 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 02:46:00.028631 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:46:00.029175 1677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:00.011946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39735 (local address 127.0.97.1:47637)
0929 02:46:00.013452 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:00.013477 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:00.013728 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:00.014340 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:00.014343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:00.015106 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:00.015652 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:00.015669 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.016764 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.016770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:00.019085 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:00.019098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.019819 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.019827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:00.020060 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:00.021033 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:00.021065 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:00.022377 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:00.025929 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:00.025937 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:00.025942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:00.026586 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:00.027073 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:00.027084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:00.027087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:00.027291 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:00.027857 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:00.027862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:00.028058 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 02:46:00.028247 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 02:46:00.028660 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"thread_start_us":865,"threads_started":1}
I20260929 02:46:00.033731 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44427
I20260929 02:46:00.033733 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44427 every 8 connection(s)
I20260929 02:46:00.034963 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:00.036996 1719 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39735
I20260929 02:46:00.037190 1719 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:00.037773 1719 heartbeater.cc:511] Master 127.0.97.62:39735 requested a full tablet report, sending...
I20260929 02:46:00.039656 1512 ts_manager.cc:194] Registered new tserver with Master: 856cc9e75d03495eb5355b04b6209630 (127.0.97.1:44427)
I20260929 02:46:00.041100 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48371
I20260929 02:46:00.043018 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 1583
I20260929 02:46:00.043323 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:00.043936 388 external_mini_cluster.cc:1428] Setting key d6ba9b5571278fe618c1cc88fa8bff38
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:00.082990 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39735
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:00.401727 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:00.402020 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:00.402182 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:00.409690 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:00.409802 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:00.409963 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:00.410085 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:00.420006 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:00.421371 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:00.422694 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:00.435549 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:00.435526 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:00.435480 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:00.436434 1723 server_base.cc:1034] running on GCE node
I20260929 02:46:00.437127 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:00.438050 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:00.439286 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790649960439208 us; error 70 us; skew 500 ppm
I20260929 02:46:00.439585 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:00 dist-test-slave-6tjh krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:00.446395 1723 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:00.448028 1723 webserver.cc:460] Webserver started at http://127.0.97.2:37261/ using document root <none> and password file <none>
I20260929 02:46:00.448838 1723 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:00.448967 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:00.449316 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:00.452230 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "c2350a618774408b90fc31f21dddb58a"
format_stamp: "Formatted at 2026-09-29 02:46:00 on dist-test-slave-6tjh"
server_key: "adbb6911c1164c8161dde76a5b5818dc"
server_key_iv: "aaefe57b56a09e64571ee5ce261cefb7"
server_key_version: "encryptionkey@0"
I20260929 02:46:00.453028 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c2350a618774408b90fc31f21dddb58a"
format_stamp: "Formatted at 2026-09-29 02:46:00 on dist-test-slave-6tjh"
server_key: "adbb6911c1164c8161dde76a5b5818dc"
server_key_iv: "aaefe57b56a09e64571ee5ce261cefb7"
server_key_version: "encryptionkey@0"
I20260929 02:46:00.458580 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:46:00.462174 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:00.463558 1723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:46:00.463742 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "c2350a618774408b90fc31f21dddb58a"
format_stamp: "Formatted at 2026-09-29 02:46:00 on dist-test-slave-6tjh"
server_key: "adbb6911c1164c8161dde76a5b5818dc"
server_key_iv: "aaefe57b56a09e64571ee5ce261cefb7"
server_key_version: "encryptionkey@0"
I20260929 02:46:00.463979 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:00.491460 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:00.496039 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:00.496338 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:00.499150 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:00.499264 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:00.499428 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:00.499512 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:46:00 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:00.524984 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:00.499853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44889 (local address 127.0.97.62:39735)
0929 02:46:00.500026 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:00.500031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:00.501524 (+ 1493us) server_negotiation.cc:407] Connection header received
0929 02:46:00.503769 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:00.503774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:00.503870 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:00.503979 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:00.506923 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.507909 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.511268 (+ 3359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.511686 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.511793 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:00.517170 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:00.517175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:00.517178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:00.517213 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:00.520946 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:00.521986 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:00.521990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:00.521993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:00.522171 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:00.522840 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:00.522844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:00.522846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:00.523116 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:00.523226 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:00.524333 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0929 02:46:00.524502 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:46:00.527292 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:00.500389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39735 (local address 127.0.97.2:44889)
0929 02:46:00.501328 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:00.501373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:00.503459 (+ 2086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:00.504418 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:00.504432 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:00.505495 (+ 1063us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:00.506702 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:00.506735 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.508091 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.508100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:00.511032 (+ 2932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:00.511052 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.511947 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.511957 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:00.512332 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:00.513234 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:00.513275 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:00.516919 (+ 3644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:00.521130 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:00.521142 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:00.521161 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:00.521818 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:00.522311 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:00.522332 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:00.522339 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:00.522687 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:00.523259 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:00.523272 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:00.523867 (+ 595us) client_negotiation.cc:769] Sending connection context
0929 02:46:00.524257 (+ 390us) client_negotiation.cc:240] Negotiation successful
0929 02:46:00.525032 (+ 775us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":446,"threads_started":1}
I20260929 02:46:00.547364 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41387
I20260929 02:46:00.547386 1857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41387 every 8 connection(s)
I20260929 02:46:00.548820 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:00.550349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 1723
I20260929 02:46:00.550570 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:00.551052 388 external_mini_cluster.cc:1428] Setting key 8791433beb3c66ab4bf7cd40717232f6
I20260929 02:46:00.552836 1858 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39735
I20260929 02:46:00.553094 1858 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:00.553424 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:00.536170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44351 (local address 127.0.97.62:39735)
0929 02:46:00.536375 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:00.536385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:00.536682 (+ 297us) server_negotiation.cc:407] Connection header received
0929 02:46:00.536959 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:00.536967 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:00.537061 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:00.537170 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:00.538861 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.539633 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.542465 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.543277 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.543413 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:00.546382 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:00.546388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:00.546392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:00.546433 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:00.549939 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:00.551104 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:00.551112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:00.551117 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:00.551263 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:00.551680 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:00.551684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:00.551687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:00.552182 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:00.552319 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:00.552851 (+ 532us) server_negotiation.cc:299] Negotiation successful
0929 02:46:00.553074 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:46:00.553540 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:00.535657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39735 (local address 127.0.97.2:44351)
0929 02:46:00.536500 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:00.536525 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:00.536808 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:00.537252 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:00.537259 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:00.538148 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:00.538686 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:00.538703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.539823 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.539832 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:00.542232 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:00.542256 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:00.543776 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:00.543787 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:00.544042 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:00.544951 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:00.544984 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:00.546171 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:00.550092 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:00.550096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:00.550100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:00.550706 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:00.551391 (+ 685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:00.551396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:00.551400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:00.551549 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:00.552302 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:00.552306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:00.552456 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 02:46:00.552678 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 02:46:00.553032 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":722,"threads_started":1}
I20260929 02:46:00.553939 1858 heartbeater.cc:511] Master 127.0.97.62:39735 requested a full tablet report, sending...
I20260929 02:46:00.555248 1512 ts_manager.cc:194] Registered new tserver with Master: c2350a618774408b90fc31f21dddb58a (127.0.97.2:41387)
I20260929 02:46:00.555976 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44889
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:00.589963 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39735
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:00.924636 1862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:00.924916 1862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:00.925014 1862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:00.932294 1862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:00.932413 1862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:00.932497 1862 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:00.932647 1862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:00.942660 1862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:39735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:00.943933 1862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:00.945361 1862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:00.959196 1870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:00.959154 1868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:00.959231 1867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:00.960073 1862 server_base.cc:1034] running on GCE node
I20260929 02:46:00.960825 1862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:00.961778 1862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:00.963027 1862 hybrid_clock.cc:648] HybridClock initialized: now 1790649960962959 us; error 45 us; skew 500 ppm
I20260929 02:46:00.963341 1862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:00 dist-test-slave-6tjh krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:00.966804 1862 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:00.968551 1862 webserver.cc:460] Webserver started at http://127.0.97.3:42697/ using document root <none> and password file <none>
I20260929 02:46:00.969383 1862 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:00.969516 1862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:00.969882 1862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:00.973690 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "81600b06281d4002ad3a6bd3f52e52e6"
format_stamp: "Formatted at 2026-09-29 02:46:00 on dist-test-slave-6tjh"
server_key: "6bf2099f3db8fde22093679884b3fff4"
server_key_iv: "ac2fe4d02296f46f838047cf8e04ab31"
server_key_version: "encryptionkey@0"
I20260929 02:46:00.974586 1862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "81600b06281d4002ad3a6bd3f52e52e6"
format_stamp: "Formatted at 2026-09-29 02:46:00 on dist-test-slave-6tjh"
server_key: "6bf2099f3db8fde22093679884b3fff4"
server_key_iv: "ac2fe4d02296f46f838047cf8e04ab31"
server_key_version: "encryptionkey@0"
I20260929 02:46:00.980261 1862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:46:00.984251 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:00.985755 1862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:00.985954 1862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "81600b06281d4002ad3a6bd3f52e52e6"
format_stamp: "Formatted at 2026-09-29 02:46:00 on dist-test-slave-6tjh"
server_key: "6bf2099f3db8fde22093679884b3fff4"
server_key_iv: "ac2fe4d02296f46f838047cf8e04ab31"
server_key_version: "encryptionkey@0"
I20260929 02:46:00.986212 1862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:01.003959 1862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:01.008315 1862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:01.008693 1862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:01.011672 1862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:01.011796 1862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:01.011992 1862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:01.012101 1862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:46:01 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:01.036799 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.012198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40969 (local address 127.0.97.62:39735)
0929 02:46:01.012398 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.012404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.013971 (+ 1567us) server_negotiation.cc:407] Connection header received
0929 02:46:01.016146 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.016156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.016264 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.016423 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:01.018786 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.020263 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.022794 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.023647 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.023813 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.028315 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:01.028321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:01.028324 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:01.028359 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:01.032141 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:01.033469 (+ 1328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:01.033475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:01.033478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:01.033639 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:01.034305 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:01.034315 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:01.034322 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:01.034860 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:01.035066 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.035828 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.036158 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:46:01.037341 1886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.012877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39735 (local address 127.0.97.3:40969)
0929 02:46:01.013770 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.013822 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.015760 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.016761 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.016773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.017646 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:01.018551 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.018586 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.020487 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.020494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.022620 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.022636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.023944 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.023955 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.024236 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.025037 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:01.025082 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:01.028068 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:01.032331 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:01.032347 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:01.032372 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:01.033294 (+ 922us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:01.033773 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:01.033781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:01.033789 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:01.034091 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:01.035027 (+ 936us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:01.035036 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:01.035389 (+ 353us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.035611 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.036021 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":396,"threads_started":1}
I20260929 02:46:01.043903 1719 heartbeater.cc:503] Master 127.0.97.62:39735 was elected leader, sending a full tablet report...
I20260929 02:46:01.070528 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.044059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50007 (local address 127.0.97.62:39735)
0929 02:46:01.044265 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.044271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.051512 (+ 7241us) server_negotiation.cc:407] Connection header received
0929 02:46:01.051814 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.051820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.051901 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.052013 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:01.053936 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.055383 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.058383 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.058956 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.059101 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.062521 (+ 3420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:01.062526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:01.062529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:01.062562 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:01.066617 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:01.067474 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:01.067479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:01.067483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:01.067722 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:01.068307 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:01.068312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:01.068316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:01.068890 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:01.069056 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.069848 (+ 792us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.070115 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:46:01.070879 1929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.044427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39735 (local address 127.0.97.3:50007)
0929 02:46:01.051382 (+ 6955us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.051404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.051582 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.052091 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.052097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.052988 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:01.053680 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.053701 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.055617 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.055623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.058165 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.058195 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.059200 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.059210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.059570 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.060672 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:01.060705 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:01.062290 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:01.066763 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:01.066767 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:01.066769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:01.067258 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:01.067868 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:01.067876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:01.067884 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:01.068134 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:01.069041 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:01.069049 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:01.069412 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.069628 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.070195 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6761,"thread_start_us":7220,"threads_started":1}
I20260929 02:46:01.076779 1862 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:36333
I20260929 02:46:01.076798 1996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:36333 every 8 connection(s)
I20260929 02:46:01.078066 1862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:01.078801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 1862
I20260929 02:46:01.078979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:01.079517 388 external_mini_cluster.cc:1428] Setting key 41d823b51792d7c80ab94db2ae99d5de
I20260929 02:46:01.080767 1997 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39735
I20260929 02:46:01.081012 1997 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:01.081769 1997 heartbeater.cc:511] Master 127.0.97.62:39735 requested a full tablet report, sending...
I20260929 02:46:01.083209 1511 ts_manager.cc:194] Registered new tserver with Master: 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3:36333)
I20260929 02:46:01.084151 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40969
I20260929 02:46:01.094738 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:46:01 dist-test-slave-6tjh krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:46:01 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:01.140681 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.126600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.0.97.62:39735)
0929 02:46:01.126829 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.126836 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.127001 (+ 165us) server_negotiation.cc:407] Connection header received
0929 02:46:01.127248 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.127253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.127316 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.127423 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:01.128909 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.129662 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.131076 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.131499 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.131635 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.135558 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:01.135563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:01.135567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:01.135603 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:01.138224 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:01.138881 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:01.138885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:01.138888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:01.139013 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:01.139321 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:01.139324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:01.139326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:01.139576 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:01.139780 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.140064 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.140233 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:46:01.144852 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:01.151804 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:01.178267 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.164089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.0.97.1:44427)
0929 02:46:01.166757 (+ 2668us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.166768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.166805 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:46:01.166929 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.166941 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.167391 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.168727 (+ 1336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:01.168987 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.171106 (+ 2119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.174956 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.176607 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.176739 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.176987 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.177786 (+ 799us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.177967 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2468,"thread_start_us":2330,"threads_started":1}
I20260929 02:46:01.178294 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.163897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44427 (local address 127.0.0.1:59260)
0929 02:46:01.165246 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.165310 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.165520 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.167701 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.167713 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.167759 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:01.168343 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.168361 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.172008 (+ 3647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.172016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.174763 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.174789 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.177141 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.177148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.177486 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.177549 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.177682 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.177872 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":84,"thread_start_us":447,"threads_started":1}
I20260929 02:46:01.180603 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.164001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.0.1:51840)
0929 02:46:01.171739 (+ 7738us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.171759 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.171848 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.173387 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.173394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.173428 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:01.173837 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.173848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.175682 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.175687 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.177652 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.177687 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.179743 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.179749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.179917 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.179948 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.180038 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.180170 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7596,"mutex_wait_us":43,"thread_start_us":7442,"threads_started":1}
I20260929 02:46:01.181321 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.164751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.0.97.3:36333)
0929 02:46:01.172724 (+ 7973us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.172732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.172772 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:46:01.172871 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.172880 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.173157 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.174020 (+ 863us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:01.174330 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.175552 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.177855 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.179540 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.179678 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.179905 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.180853 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.181017 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7762,"thread_start_us":7604,"threads_started":1}
I20260929 02:46:01.181609 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.165596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46946 (local address 127.0.97.2:41387)
0929 02:46:01.166260 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.166272 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.173117 (+ 6845us) server_negotiation.cc:407] Connection header received
0929 02:46:01.173453 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.173461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.173700 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.173876 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:01.174632 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.176036 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.178842 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.180210 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.180337 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.180569 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.181126 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.181297 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":256,"threads_started":1}
I20260929 02:46:01.182148 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.165041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41387 (local address 127.0.0.1:46946)
0929 02:46:01.172989 (+ 7948us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.173012 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.173230 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.174091 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.174097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.174124 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:01.174461 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.174468 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.176208 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.176213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.178615 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.178631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.180725 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.180730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.180920 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.180953 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.181830 (+ 877us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.181930 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7819,"thread_start_us":7660,"threads_started":1}
I20260929 02:46:01.185534 1788 tablet_service.cc:1467] Processing CreateTablet for tablet a679760a801a42809c44bbaec57cb395 (DEFAULT_TABLE table=test-table [id=dc7cdf0676694e828738947f3bb8062b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:01.185513 1932 tablet_service.cc:1467] Processing CreateTablet for tablet a679760a801a42809c44bbaec57cb395 (DEFAULT_TABLE table=test-table [id=dc7cdf0676694e828738947f3bb8062b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:01.186720 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679760a801a42809c44bbaec57cb395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:01.186730 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679760a801a42809c44bbaec57cb395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:01.182601 1649 tablet_service.cc:1467] Processing CreateTablet for tablet a679760a801a42809c44bbaec57cb395 (DEFAULT_TABLE table=test-table [id=dc7cdf0676694e828738947f3bb8062b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:01.191234 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a679760a801a42809c44bbaec57cb395. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:01.197934 2016 tablet_bootstrap.cc:492] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Bootstrap starting.
I20260929 02:46:01.200399 2017 tablet_bootstrap.cc:492] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Bootstrap starting.
I20260929 02:46:01.201570 2016 tablet_bootstrap.cc:654] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:01.203323 2016 log.cc:824] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:01.205305 2017 tablet_bootstrap.cc:654] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:01.206535 2017 log.cc:824] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:01.206739 2016 tablet_bootstrap.cc:492] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: No bootstrap required, opened a new log
I20260929 02:46:01.207029 2016 ts_tablet_manager.cc:1397] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:46:01.207064 2019 tablet_bootstrap.cc:492] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Bootstrap starting.
I20260929 02:46:01.209786 2017 tablet_bootstrap.cc:492] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: No bootstrap required, opened a new log
I20260929 02:46:01.210067 2017 ts_tablet_manager.cc:1397] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:46:01.210172 2019 tablet_bootstrap.cc:654] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:01.211568 2019 log.cc:824] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:01.217772 2016 raft_consensus.cc:348] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.218278 2016 raft_consensus.cc:374] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:01.218379 2016 raft_consensus.cc:729] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2350a618774408b90fc31f21dddb58a, State: Initialized, Role: FOLLOWER
I20260929 02:46:01.219082 2016 consensus_queue.cc:260] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.220302 2019 tablet_bootstrap.cc:492] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: No bootstrap required, opened a new log
I20260929 02:46:01.220654 2019 ts_tablet_manager.cc:1397] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260929 02:46:01.220788 1858 heartbeater.cc:503] Master 127.0.97.62:39735 was elected leader, sending a full tablet report...
I20260929 02:46:01.221411 2016 ts_tablet_manager.cc:1428] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20260929 02:46:01.221413 2017 raft_consensus.cc:348] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.221840 2017 raft_consensus.cc:374] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:01.221948 2017 raft_consensus.cc:729] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81600b06281d4002ad3a6bd3f52e52e6, State: Initialized, Role: FOLLOWER
I20260929 02:46:01.222586 2017 consensus_queue.cc:260] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.223660 1997 heartbeater.cc:503] Master 127.0.97.62:39735 was elected leader, sending a full tablet report...
I20260929 02:46:01.224329 2017 ts_tablet_manager.cc:1428] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 02:46:01.230252 2019 raft_consensus.cc:348] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.230650 2019 raft_consensus.cc:374] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:01.230726 2019 raft_consensus.cc:729] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856cc9e75d03495eb5355b04b6209630, State: Initialized, Role: FOLLOWER
I20260929 02:46:01.231204 2019 consensus_queue.cc:260] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.232872 2019 ts_tablet_manager.cc:1428] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
W20260929 02:46:01.287211 1720 tablet.cc:2369] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:46:01.301753 1859 tablet.cc:2369] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:46:01.329828 1998 tablet.cc:2369] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:01.498456 2022 raft_consensus.cc:482] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:01.498786 2022 raft_consensus.cc:504] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.500576 2022 leader_election.cc:288] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 856cc9e75d03495eb5355b04b6209630 (127.0.97.1:44427), 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3:36333)
I20260929 02:46:01.508142 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.501070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59661 (local address 127.0.97.1:44427)
0929 02:46:01.501314 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.501323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.501999 (+ 676us) server_negotiation.cc:407] Connection header received
0929 02:46:01.502179 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.502185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.502262 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.502354 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:01.503027 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.504347 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.506357 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.507134 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.507291 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.507398 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.507739 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.507876 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:46:01.508165 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.500984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55905 (local address 127.0.97.3:36333)
0929 02:46:01.501169 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.501178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.502183 (+ 1005us) server_negotiation.cc:407] Connection header received
0929 02:46:01.502392 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.502397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.502472 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.502572 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:01.503679 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.504351 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.506400 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.507137 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.507278 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.507398 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.507799 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.507922 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:46:01.508285 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.500951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44427 (local address 127.0.97.2:59661)
0929 02:46:01.501898 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.501916 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.502038 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.502383 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.502388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.502411 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:01.502880 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.502890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.504479 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.504483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.506205 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.506216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.507383 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.507389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.507549 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.507613 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.507683 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.507833 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":47,"spinlock_wait_cycles":768,"thread_start_us":792,"threads_started":1}
I20260929 02:46:01.508796 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.500978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.97.2:55905)
0929 02:46:01.502061 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.502077 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.502228 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.502595 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.502599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.502616 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:01.503510 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.503524 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.504508 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.504514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.506221 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.506233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.507370 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.507376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.507623 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.507647 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.507755 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.507875 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":123,"thread_start_us":402,"threads_started":1}
I20260929 02:46:01.509438 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679760a801a42809c44bbaec57cb395" candidate_uuid: "c2350a618774408b90fc31f21dddb58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856cc9e75d03495eb5355b04b6209630" is_pre_election: true
I20260929 02:46:01.509786 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679760a801a42809c44bbaec57cb395" candidate_uuid: "c2350a618774408b90fc31f21dddb58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81600b06281d4002ad3a6bd3f52e52e6" is_pre_election: true
I20260929 02:46:01.509944 1670 raft_consensus.cc:2454] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2350a618774408b90fc31f21dddb58a in term 0.
I20260929 02:46:01.510228 1952 raft_consensus.cc:2454] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2350a618774408b90fc31f21dddb58a in term 0.
I20260929 02:46:01.510701 1742 leader_election.cc:302] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 856cc9e75d03495eb5355b04b6209630, c2350a618774408b90fc31f21dddb58a; no voters:
I20260929 02:46:01.511090 2022 raft_consensus.cc:2781] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:01.511211 2022 raft_consensus.cc:482] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:01.511284 2022 raft_consensus.cc:3037] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:01.513446 2022 raft_consensus.cc:504] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.514003 2022 leader_election.cc:288] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [CANDIDATE]: Term 1 election: Requested vote from peers 856cc9e75d03495eb5355b04b6209630 (127.0.97.1:44427), 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3:36333)
I20260929 02:46:01.514459 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679760a801a42809c44bbaec57cb395" candidate_uuid: "c2350a618774408b90fc31f21dddb58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856cc9e75d03495eb5355b04b6209630"
I20260929 02:46:01.514479 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a679760a801a42809c44bbaec57cb395" candidate_uuid: "c2350a618774408b90fc31f21dddb58a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81600b06281d4002ad3a6bd3f52e52e6"
I20260929 02:46:01.514669 1952 raft_consensus.cc:3037] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:01.514685 1670 raft_consensus.cc:3037] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:01.516778 1952 raft_consensus.cc:2454] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2350a618774408b90fc31f21dddb58a in term 1.
I20260929 02:46:01.517199 1740 leader_election.cc:302] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81600b06281d4002ad3a6bd3f52e52e6, c2350a618774408b90fc31f21dddb58a; no voters:
I20260929 02:46:01.517395 1670 raft_consensus.cc:2454] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2350a618774408b90fc31f21dddb58a in term 1.
I20260929 02:46:01.517530 2022 raft_consensus.cc:2781] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:01.517990 2022 raft_consensus.cc:686] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 1 LEADER]: Becoming Leader. State: Replica: c2350a618774408b90fc31f21dddb58a, State: Running, Role: LEADER
I20260929 02:46:01.518460 2022 consensus_queue.cc:237] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } }
I20260929 02:46:01.523322 1511 catalog_manager.cc:5495] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a reported cstate change: term changed from 0 to 1, leader changed from <none> to c2350a618774408b90fc31f21dddb58a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "c2350a618774408b90fc31f21dddb58a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } health_report { overall_health: HEALTHY } } }
I20260929 02:46:01.560261 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.553563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.0.97.2:41387)
0929 02:46:01.553769 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.553774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.553802 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:46:01.554171 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.554176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.554313 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.554430 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:01.554840 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.555684 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.557794 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.558264 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.558382 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.558995 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:01.559560 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:01.559685 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.559837 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.559910 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:46:01.568950 1952 raft_consensus.cc:1261] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Refusing update from remote peer c2350a618774408b90fc31f21dddb58a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:01.568953 1670 raft_consensus.cc:1261] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Refusing update from remote peer c2350a618774408b90fc31f21dddb58a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:01.570056 2027 consensus_queue.cc:1035] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [LEADER]: Connected to new peer: Peer: permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:01.570477 2022 consensus_queue.cc:1035] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [LEADER]: Connected to new peer: Peer: permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:01.601614 1512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:01.601919 1512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:01.605381 1512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1: Sending DeleteTablet for 3 replicas of tablet a679760a801a42809c44bbaec57cb395
I20260929 02:46:01.606542 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet a679760a801a42809c44bbaec57cb395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59260
I20260929 02:46:01.606633 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet a679760a801a42809c44bbaec57cb395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46946
I20260929 02:46:01.607573 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet a679760a801a42809c44bbaec57cb395 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:01 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51840
I20260929 02:46:01.610406 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:01.611261 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:01.618372 2042 tablet_replica.cc:330] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: stopping tablet replica
I20260929 02:46:01.618788 2042 raft_consensus.cc:2229] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:01.619315 2042 raft_consensus.cc:2258] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:01.621670 2040 tablet_replica.cc:330] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: stopping tablet replica
I20260929 02:46:01.622159 2040 raft_consensus.cc:2229] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:01.622225 2041 tablet_replica.cc:330] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: stopping tablet replica
I20260929 02:46:01.622674 2040 raft_consensus.cc:2258] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:01.622648 2041 raft_consensus.cc:2229] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:01.623915 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4cedb3fdf84f079dfdad5a9d08070c (DEFAULT_TABLE table=test-table [id=8efb21e67bca4a3ca67de3e59605248e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:01.623920 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4cedb3fdf84f079dfdad5a9d08070c (DEFAULT_TABLE table=test-table [id=8efb21e67bca4a3ca67de3e59605248e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:01.624590 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4cedb3fdf84f079dfdad5a9d08070c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:01.624639 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4cedb3fdf84f079dfdad5a9d08070c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:01.629139 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 7a4cedb3fdf84f079dfdad5a9d08070c (DEFAULT_TABLE table=test-table [id=8efb21e67bca4a3ca67de3e59605248e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:01.629796 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a4cedb3fdf84f079dfdad5a9d08070c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:01.630479 2019 tablet_bootstrap.cc:492] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: Bootstrap starting.
I20260929 02:46:01.632171 2019 tablet_bootstrap.cc:654] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:01.632666 2016 tablet_bootstrap.cc:492] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: Bootstrap starting.
I20260929 02:46:01.634670 2016 tablet_bootstrap.cc:654] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:01.623389 2041 raft_consensus.cc:2258] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:01.638067 2041 ts_tablet_manager.cc:1905] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:01.640046 2042 ts_tablet_manager.cc:1905] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:01.640703 2040 ts_tablet_manager.cc:1905] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:01.641366 2017 tablet_bootstrap.cc:492] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: Bootstrap starting.
I20260929 02:46:01.642953 2017 tablet_bootstrap.cc:654] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:01.650923 2040 ts_tablet_manager.cc:1918] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:01.651178 2040 log.cc:1196] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/a679760a801a42809c44bbaec57cb395
I20260929 02:46:01.651278 2016 tablet_bootstrap.cc:492] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: No bootstrap required, opened a new log
I20260929 02:46:01.651454 2016 ts_tablet_manager.cc:1397] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20260929 02:46:01.651660 2019 tablet_bootstrap.cc:492] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: No bootstrap required, opened a new log
I20260929 02:46:01.651820 2019 ts_tablet_manager.cc:1397] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.008s
I20260929 02:46:01.652185 2040 ts_tablet_manager.cc:1939] T a679760a801a42809c44bbaec57cb395 P 856cc9e75d03495eb5355b04b6209630: Deleting consensus metadata
I20260929 02:46:01.652318 2016 raft_consensus.cc:348] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.652596 2016 raft_consensus.cc:374] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:01.652729 2016 raft_consensus.cc:729] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2350a618774408b90fc31f21dddb58a, State: Initialized, Role: FOLLOWER
I20260929 02:46:01.652794 2019 raft_consensus.cc:348] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.653005 2016 consensus_queue.cc:260] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.653036 2019 raft_consensus.cc:374] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:01.653355 2019 raft_consensus.cc:729] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856cc9e75d03495eb5355b04b6209630, State: Initialized, Role: FOLLOWER
I20260929 02:46:01.654942 1499 catalog_manager.cc:4841] TS 856cc9e75d03495eb5355b04b6209630 (127.0.97.1:44427): tablet a679760a801a42809c44bbaec57cb395 (table test-table [id=dc7cdf0676694e828738947f3bb8062b]) successfully deleted
I20260929 02:46:01.655242 2016 ts_tablet_manager.cc:1428] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260929 02:46:01.657930 2041 ts_tablet_manager.cc:1918] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:01.658099 2041 log.cc:1196] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/a679760a801a42809c44bbaec57cb395
I20260929 02:46:01.658665 2041 ts_tablet_manager.cc:1939] T a679760a801a42809c44bbaec57cb395 P c2350a618774408b90fc31f21dddb58a: Deleting consensus metadata
I20260929 02:46:01.658805 2042 ts_tablet_manager.cc:1918] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:01.659041 2042 log.cc:1196] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/a679760a801a42809c44bbaec57cb395
I20260929 02:46:01.659650 2042 ts_tablet_manager.cc:1939] T a679760a801a42809c44bbaec57cb395 P 81600b06281d4002ad3a6bd3f52e52e6: Deleting consensus metadata
I20260929 02:46:01.660202 2017 tablet_bootstrap.cc:492] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: No bootstrap required, opened a new log
I20260929 02:46:01.660365 2017 ts_tablet_manager.cc:1397] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260929 02:46:01.661113 1496 catalog_manager.cc:4841] TS c2350a618774408b90fc31f21dddb58a (127.0.97.2:41387): tablet a679760a801a42809c44bbaec57cb395 (table test-table [id=dc7cdf0676694e828738947f3bb8062b]) successfully deleted
I20260929 02:46:01.654430 2019 consensus_queue.cc:260] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.661652 1497 catalog_manager.cc:4841] TS 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3:36333): tablet a679760a801a42809c44bbaec57cb395 (table test-table [id=dc7cdf0676694e828738947f3bb8062b]) successfully deleted
I20260929 02:46:01.662106 2017 raft_consensus.cc:348] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.662355 2017 raft_consensus.cc:374] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:01.662460 2017 raft_consensus.cc:729] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81600b06281d4002ad3a6bd3f52e52e6, State: Initialized, Role: FOLLOWER
I20260929 02:46:01.662704 2019 ts_tablet_manager.cc:1428] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260929 02:46:01.662773 2017 consensus_queue.cc:260] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.664669 2017 ts_tablet_manager.cc:1428] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260929 02:46:01.925055 2024 raft_consensus.cc:482] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:01.925340 2024 raft_consensus.cc:504] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.926973 2024 leader_election.cc:288] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2350a618774408b90fc31f21dddb58a (127.0.97.2:41387), 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3:36333)
I20260929 02:46:01.936662 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.928109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50163 (local address 127.0.97.3:36333)
0929 02:46:01.928321 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.928329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.929183 (+ 854us) server_negotiation.cc:407] Connection header received
0929 02:46:01.929495 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.929502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.929591 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.929660 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:01.930548 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.931525 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.934328 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.935550 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.935694 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.935805 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.936204 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.936343 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:46:01.936762 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.927473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.97.1:50163)
0929 02:46:01.929074 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.929093 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.929301 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.929870 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.929874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.929900 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:01.930331 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.930347 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.931644 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.931651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.934179 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.934192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.935782 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.935791 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.935976 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.936018 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.936171 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.936337 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1455,"mutex_wait_us":93,"thread_start_us":1044,"threads_started":1}
I20260929 02:46:01.936910 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.927784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47479 (local address 127.0.97.2:41387)
0929 02:46:01.927948 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:01.927956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:01.929008 (+ 1052us) server_negotiation.cc:407] Connection header received
0929 02:46:01.929233 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:01.929237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:01.929298 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:01.929397 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:01.929923 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.931107 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.934440 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.935706 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.935845 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.935919 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:01.936443 (+ 524us) server_negotiation.cc:299] Negotiation successful
0929 02:46:01.936609 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 02:46:01.937541 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:01.927221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41387 (local address 127.0.97.1:47479)
0929 02:46:01.928879 (+ 1658us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:01.928902 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:01.929074 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:01.929434 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:01.929441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:01.929467 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:01.929752 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.929782 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.931260 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.931266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.934192 (+ 2926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:01.934207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:01.936067 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:01.936071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:01.936261 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:01.936293 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:46:01.937092 (+ 799us) client_negotiation.cc:240] Negotiation successful
0929 02:46:01.937236 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":42,"thread_start_us":1544,"threads_started":1}
I20260929 02:46:01.937808 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4cedb3fdf84f079dfdad5a9d08070c" candidate_uuid: "856cc9e75d03495eb5355b04b6209630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81600b06281d4002ad3a6bd3f52e52e6" is_pre_election: true
I20260929 02:46:01.938081 1952 raft_consensus.cc:2454] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856cc9e75d03495eb5355b04b6209630 in term 0.
I20260929 02:46:01.938482 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4cedb3fdf84f079dfdad5a9d08070c" candidate_uuid: "856cc9e75d03495eb5355b04b6209630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2350a618774408b90fc31f21dddb58a" is_pre_election: true
I20260929 02:46:01.938616 1600 leader_election.cc:302] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81600b06281d4002ad3a6bd3f52e52e6, 856cc9e75d03495eb5355b04b6209630; no voters:
I20260929 02:46:01.938957 1810 raft_consensus.cc:2454] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 856cc9e75d03495eb5355b04b6209630 in term 0.
I20260929 02:46:01.939043 2024 raft_consensus.cc:2781] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:01.939154 2024 raft_consensus.cc:482] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:01.939289 2024 raft_consensus.cc:3037] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:01.942201 2024 raft_consensus.cc:504] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.942870 2024 leader_election.cc:288] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [CANDIDATE]: Term 1 election: Requested vote from peers c2350a618774408b90fc31f21dddb58a (127.0.97.2:41387), 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3:36333)
I20260929 02:46:01.943446 1952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4cedb3fdf84f079dfdad5a9d08070c" candidate_uuid: "856cc9e75d03495eb5355b04b6209630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81600b06281d4002ad3a6bd3f52e52e6"
I20260929 02:46:01.943684 1952 raft_consensus.cc:3037] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:01.946575 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a4cedb3fdf84f079dfdad5a9d08070c" candidate_uuid: "856cc9e75d03495eb5355b04b6209630" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2350a618774408b90fc31f21dddb58a"
I20260929 02:46:01.946762 1952 raft_consensus.cc:2454] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856cc9e75d03495eb5355b04b6209630 in term 1.
I20260929 02:46:01.947042 1810 raft_consensus.cc:3037] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:01.947378 1600 leader_election.cc:302] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81600b06281d4002ad3a6bd3f52e52e6, 856cc9e75d03495eb5355b04b6209630; no voters:
I20260929 02:46:01.947711 2024 raft_consensus.cc:2781] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:01.949287 2024 raft_consensus.cc:686] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 1 LEADER]: Becoming Leader. State: Replica: 856cc9e75d03495eb5355b04b6209630, State: Running, Role: LEADER
I20260929 02:46:01.949810 2024 consensus_queue.cc:237] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:01.951112 1810 raft_consensus.cc:2454] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 856cc9e75d03495eb5355b04b6209630 in term 1.
I20260929 02:46:01.954344 1511 catalog_manager.cc:5495] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 reported cstate change: term changed from 0 to 1, leader changed from <none> to 856cc9e75d03495eb5355b04b6209630 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "856cc9e75d03495eb5355b04b6209630" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: UNKNOWN } } }
Sep 29 02:46:02 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649959, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:02.063179 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.035007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39735 (local address 127.0.0.1:55552)
0929 02:46:02.048848 (+ 13841us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:02.048872 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:02.049020 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:02.049635 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:02.049643 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:02.050404 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:02.050600 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.050610 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.051710 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.051720 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.052747 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.052763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.053735 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.053741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.054375 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.055193 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:02.055210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:02.057769 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:02.060586 (+ 2817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:02.060598 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:02.060608 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:02.061039 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:02.061402 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:02.061408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:02.061411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:02.061549 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:02.062044 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:02.062052 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:02.062226 (+ 174us) client_negotiation.cc:769] Sending connection context
0929 02:46:02.062391 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 02:46:02.062685 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13662,"thread_start_us":13444,"threads_started":1}
I20260929 02:46:02.063197 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.035007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55552 (local address 127.0.97.62:39735)
0929 02:46:02.049313 (+ 14306us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.049317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.049338 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:46:02.049399 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.049402 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.049456 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.049601 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:02.050767 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.051552 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.052914 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.053543 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.053715 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.058007 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:02.058012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:02.058015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:02.058044 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:02.060439 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.061163 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.061170 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.061178 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.061287 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.061657 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.061662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.061665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.061920 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:02.062067 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.062472 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.062788 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":14188,"thread_start_us":14032,"threads_started":1}
I20260929 02:46:02.068235 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55552:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:46:02.069271 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:02.076303 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 1fabb47b4e914e55bb3f7816e435fc3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5af1280e2030405e8872caaec0f3823e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:02.076361 1787 tablet_service.cc:1467] Processing CreateTablet for tablet 1fabb47b4e914e55bb3f7816e435fc3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5af1280e2030405e8872caaec0f3823e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:02.076943 1787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabb47b4e914e55bb3f7816e435fc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:02.077066 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 1fabb47b4e914e55bb3f7816e435fc3c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5af1280e2030405e8872caaec0f3823e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:02.077601 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabb47b4e914e55bb3f7816e435fc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:02.082167 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fabb47b4e914e55bb3f7816e435fc3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:02.084578 2016 tablet_bootstrap.cc:492] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: Bootstrap starting.
I20260929 02:46:02.086303 2016 tablet_bootstrap.cc:654] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:02.086408 2017 tablet_bootstrap.cc:492] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: Bootstrap starting.
I20260929 02:46:02.088284 2017 tablet_bootstrap.cc:654] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:02.098115 2019 tablet_bootstrap.cc:492] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: Bootstrap starting.
I20260929 02:46:02.099366 2017 tablet_bootstrap.cc:492] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: No bootstrap required, opened a new log
I20260929 02:46:02.099574 2017 ts_tablet_manager.cc:1397] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20260929 02:46:02.100437 2016 tablet_bootstrap.cc:492] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: No bootstrap required, opened a new log
I20260929 02:46:02.100473 2019 tablet_bootstrap.cc:654] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:02.100481 2017 raft_consensus.cc:348] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.100785 2016 ts_tablet_manager.cc:1397] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20260929 02:46:02.100790 2017 raft_consensus.cc:374] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:02.100896 2017 raft_consensus.cc:729] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81600b06281d4002ad3a6bd3f52e52e6, State: Initialized, Role: FOLLOWER
I20260929 02:46:02.101245 2017 consensus_queue.cc:260] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.101681 2016 raft_consensus.cc:348] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.101886 2017 ts_tablet_manager.cc:1428] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 02:46:02.102012 2016 raft_consensus.cc:374] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:02.101783 2023 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.102200 2023 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:02.102202 2016 raft_consensus.cc:729] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2350a618774408b90fc31f21dddb58a, State: Initialized, Role: FOLLOWER
I20260929 02:46:02.102728 2016 consensus_queue.cc:260] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.103319 2016 ts_tablet_manager.cc:1428] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:46:02.103181 2027 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.104297 2027 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:02.106118 2019 tablet_bootstrap.cc:492] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: No bootstrap required, opened a new log
I20260929 02:46:02.106323 2019 ts_tablet_manager.cc:1397] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260929 02:46:02.107138 2019 raft_consensus.cc:348] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.107410 2019 raft_consensus.cc:374] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:02.107476 2019 raft_consensus.cc:729] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 856cc9e75d03495eb5355b04b6209630, State: Initialized, Role: FOLLOWER
I20260929 02:46:02.107662 2019 consensus_queue.cc:260] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.108240 2019 ts_tablet_manager.cc:1428] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:46:02.108090 2024 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.108419 2024 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:02.253854 2023 raft_consensus.cc:482] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:02.254048 2023 raft_consensus.cc:504] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.255604 2023 leader_election.cc:288] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 856cc9e75d03495eb5355b04b6209630 (127.0.97.1:44427), c2350a618774408b90fc31f21dddb58a (127.0.97.2:41387)
I20260929 02:46:02.264516 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.256007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50063 (local address 127.0.97.2:41387)
0929 02:46:02.256197 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.256203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.256783 (+ 580us) server_negotiation.cc:407] Connection header received
0929 02:46:02.256882 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.256891 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.256973 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.257064 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:02.257696 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.259156 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.262113 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.263387 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.263526 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.263635 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.264084 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.264243 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:46:02.264539 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.256024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44799 (local address 127.0.97.1:44427)
0929 02:46:02.256506 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.256511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.256561 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:46:02.256728 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.256732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.256852 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.256971 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:02.257602 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.259170 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.262113 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.263387 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.263526 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.263643 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.264083 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.264224 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":303,"threads_started":1}
I20260929 02:46:02.264838 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.255869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41387 (local address 127.0.97.3:50063)
0929 02:46:02.256677 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:02.256693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:02.256788 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:02.257090 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:02.257095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:02.257161 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:02.257530 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.257541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.259322 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.259325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.261886 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.261903 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.263618 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.263623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.263876 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.263944 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 02:46:02.264045 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:46:02.264222 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"mutex_wait_us":471,"spinlock_wait_cycles":1408,"thread_start_us":236,"threads_started":1}
I20260929 02:46:02.264874 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.255869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44427 (local address 127.0.97.3:44799)
0929 02:46:02.256354 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:02.256381 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:02.256530 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:02.256994 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:02.256997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:02.257015 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:02.257409 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.257423 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.259319 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.259325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.261887 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.261903 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.263583 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.263591 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.263868 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.263944 (+ 76us) client_negotiation.cc:769] Sending connection context
0929 02:46:02.264045 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 02:46:02.264222 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":298,"spinlock_wait_cycles":9472,"thread_start_us":273,"threads_started":1}
I20260929 02:46:02.265543 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabb47b4e914e55bb3f7816e435fc3c" candidate_uuid: "81600b06281d4002ad3a6bd3f52e52e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856cc9e75d03495eb5355b04b6209630" is_pre_election: true
I20260929 02:46:02.265558 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabb47b4e914e55bb3f7816e435fc3c" candidate_uuid: "81600b06281d4002ad3a6bd3f52e52e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2350a618774408b90fc31f21dddb58a" is_pre_election: true
I20260929 02:46:02.265793 1670 raft_consensus.cc:2454] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81600b06281d4002ad3a6bd3f52e52e6 in term 0.
I20260929 02:46:02.265787 1810 raft_consensus.cc:2454] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81600b06281d4002ad3a6bd3f52e52e6 in term 0.
I20260929 02:46:02.266273 1881 leader_election.cc:302] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 81600b06281d4002ad3a6bd3f52e52e6, 856cc9e75d03495eb5355b04b6209630, c2350a618774408b90fc31f21dddb58a; no voters:
I20260929 02:46:02.266908 2023 raft_consensus.cc:2781] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:02.267041 2023 raft_consensus.cc:482] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:02.267158 2023 raft_consensus.cc:3037] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:02.269580 2023 raft_consensus.cc:504] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.270237 2023 leader_election.cc:288] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [CANDIDATE]: Term 1 election: Requested vote from peers 856cc9e75d03495eb5355b04b6209630 (127.0.97.1:44427), c2350a618774408b90fc31f21dddb58a (127.0.97.2:41387)
I20260929 02:46:02.270716 1670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabb47b4e914e55bb3f7816e435fc3c" candidate_uuid: "81600b06281d4002ad3a6bd3f52e52e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "856cc9e75d03495eb5355b04b6209630"
I20260929 02:46:02.270758 1810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fabb47b4e914e55bb3f7816e435fc3c" candidate_uuid: "81600b06281d4002ad3a6bd3f52e52e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2350a618774408b90fc31f21dddb58a"
I20260929 02:46:02.270939 1670 raft_consensus.cc:3037] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:02.270968 1810 raft_consensus.cc:3037] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:02.272945 1810 raft_consensus.cc:2454] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81600b06281d4002ad3a6bd3f52e52e6 in term 1.
I20260929 02:46:02.273051 1670 raft_consensus.cc:2454] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81600b06281d4002ad3a6bd3f52e52e6 in term 1.
I20260929 02:46:02.273483 1878 leader_election.cc:302] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81600b06281d4002ad3a6bd3f52e52e6, c2350a618774408b90fc31f21dddb58a; no voters:
I20260929 02:46:02.273823 2023 raft_consensus.cc:2781] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:02.274206 2023 raft_consensus.cc:686] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 LEADER]: Becoming Leader. State: Replica: 81600b06281d4002ad3a6bd3f52e52e6, State: Running, Role: LEADER
I20260929 02:46:02.274636 2023 consensus_queue.cc:237] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } }
I20260929 02:46:02.277019 2063 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: TxnStatusTablet state changed. Reason: New leader 81600b06281d4002ad3a6bd3f52e52e6. Latest consensus state: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.277427 2063 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:02.278141 2065 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:46:02.278357 2065 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:46:02.278857 1511 catalog_manager.cc:5495] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } health_report { overall_health: HEALTHY } } }
I20260929 02:46:02.331434 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.325043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.0.1:51854)
0929 02:46:02.325320 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:02.325345 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:02.325477 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:02.325988 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:02.325993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:02.326006 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:02.326254 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.326269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.327487 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.327495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.328662 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.328674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.329564 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.329571 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.329811 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.329916 (+ 105us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:02.330909 (+ 993us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:02.330933 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:46:02.331029 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:46:02.331177 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 02:46:02.331477 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.325247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51854 (local address 127.0.97.3:36333)
0929 02:46:02.325402 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.325407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.325425 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:46:02.325712 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.325717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.325803 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.325973 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:02.326435 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.327353 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.328816 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.329393 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.329551 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.330132 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:02.330778 (+ 646us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:02.330944 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.331072 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.331164 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:46:02.336551 1670 raft_consensus.cc:1261] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Refusing update from remote peer 81600b06281d4002ad3a6bd3f52e52e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:02.336910 1810 raft_consensus.cc:1261] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Refusing update from remote peer 81600b06281d4002ad3a6bd3f52e52e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:02.337387 2063 consensus_queue.cc:1035] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:02.337742 2023 consensus_queue.cc:1035] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:02.348390 2027 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: TxnStatusTablet state changed. Reason: New leader 81600b06281d4002ad3a6bd3f52e52e6. Latest consensus state: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.350123 2027 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:02.353839 2070 mvcc.cc:205] Tried to move back new op lower bound from 7334502245722988544 to 7334502245482213376. Current Snapshot: MvccSnapshot[applied={T|T < 7334502245722988544}]
I20260929 02:46:02.357674 2027 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.358134 2027 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P c2350a618774408b90fc31f21dddb58a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:02.360847 2024 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: TxnStatusTablet state changed. Reason: New leader 81600b06281d4002ad3a6bd3f52e52e6. Latest consensus state: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.361240 2024 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:02.366896 2063 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.367221 2063 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:02.367836 2048 consensus_queue.cc:1035] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:02.368821 2063 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.369079 2063 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P 81600b06281d4002ad3a6bd3f52e52e6: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:02.370888 2024 tablet_replica.cc:439] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "81600b06281d4002ad3a6bd3f52e52e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "856cc9e75d03495eb5355b04b6209630" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44427 } } peers { permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 } } peers { permanent_uuid: "81600b06281d4002ad3a6bd3f52e52e6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 36333 } } }
I20260929 02:46:02.371306 2024 tablet_replica.cc:442] T 1fabb47b4e914e55bb3f7816e435fc3c P 856cc9e75d03495eb5355b04b6209630: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:02.377774 2048 consensus_queue.cc:1035] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2350a618774408b90fc31f21dddb58a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:02.392537 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.385319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.0.97.1:44427)
0929 02:46:02.385514 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.385519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.386236 (+ 717us) server_negotiation.cc:407] Connection header received
0929 02:46:02.386527 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.386533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.386636 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.386808 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:02.387418 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.388526 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.389963 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.390638 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.390824 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.391229 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:02.391969 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:02.392116 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.392175 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.392263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:46:02.410746 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.403419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40961 (local address 127.0.97.3:36333)
0929 02:46:02.403625 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.403632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.404285 (+ 653us) server_negotiation.cc:407] Connection header received
0929 02:46:02.404491 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.404498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.404618 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.404724 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:02.405248 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.406407 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.408394 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.409174 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.409302 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.409849 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:02.409989 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:02.410081 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.410287 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.410374 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:46:02.410784 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.403430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:36333 (local address 127.0.97.1:40961)
0929 02:46:02.404170 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:02.404188 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:02.404329 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:02.404739 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:02.404744 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:02.404757 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:02.405109 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.405119 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.406543 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.406549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.408230 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.408245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.409367 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.409376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.409565 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.409646 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:02.410092 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:02.410104 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 02:46:02.410249 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 02:46:02.410405 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":385,"threads_started":1}
I20260929 02:46:02.454903 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.448512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44013 (local address 127.0.97.1:44427)
0929 02:46:02.448693 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.448708 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.448951 (+ 243us) server_negotiation.cc:407] Connection header received
0929 02:46:02.449137 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.449142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.449221 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.449318 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:02.449788 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.450885 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.452804 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.453477 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.453602 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.454088 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:02.454273 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:02.454359 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.454547 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.454624 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:46:02.455525 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.448157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44427 (local address 127.0.97.3:44013)
0929 02:46:02.448844 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:02.448864 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:02.448991 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:02.449335 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:02.449338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:02.449358 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:02.449661 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.449668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.450996 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.451003 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.452640 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:02.452653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.453691 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.453695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:02.453841 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.453923 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:02.454408 (+ 485us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:02.454419 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:46:02.454500 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:46:02.454850 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":559,"threads_started":1}
I20260929 02:46:02.492780 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:02.493037 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:02.496598 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65435bd814ba47bdb3b921bfa615a1c1: Sending DeleteTablet for 3 replicas of tablet 7a4cedb3fdf84f079dfdad5a9d08070c
I20260929 02:46:02.497447 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4cedb3fdf84f079dfdad5a9d08070c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59260
I20260929 02:46:02.497797 1787 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4cedb3fdf84f079dfdad5a9d08070c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46946
I20260929 02:46:02.499859 1932 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a4cedb3fdf84f079dfdad5a9d08070c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51840
I20260929 02:46:02.508826 2089 tablet_replica.cc:330] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: stopping tablet replica
I20260929 02:46:02.509608 2089 raft_consensus.cc:2229] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:02.510761 2090 tablet_replica.cc:330] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: stopping tablet replica
I20260929 02:46:02.511173 2090 raft_consensus.cc:2229] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:02.511435 2089 raft_consensus.cc:2258] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:02.512754 2090 raft_consensus.cc:2258] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:02.513150 2089 ts_tablet_manager.cc:1905] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:02.513907 2090 ts_tablet_manager.cc:1905] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:02.522539 2091 tablet_replica.cc:330] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: stopping tablet replica
I20260929 02:46:02.522781 2091 raft_consensus.cc:2229] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:02.523064 2091 raft_consensus.cc:2258] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:02.524485 2091 ts_tablet_manager.cc:1905] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:02.525084 2089 ts_tablet_manager.cc:1918] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:02.529412 2089 log.cc:1196] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/7a4cedb3fdf84f079dfdad5a9d08070c
I20260929 02:46:02.530043 2090 ts_tablet_manager.cc:1918] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:02.530066 2089 ts_tablet_manager.cc:1939] T 7a4cedb3fdf84f079dfdad5a9d08070c P 856cc9e75d03495eb5355b04b6209630: Deleting consensus metadata
I20260929 02:46:02.530185 2090 log.cc:1196] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/7a4cedb3fdf84f079dfdad5a9d08070c
I20260929 02:46:02.530622 2090 ts_tablet_manager.cc:1939] T 7a4cedb3fdf84f079dfdad5a9d08070c P c2350a618774408b90fc31f21dddb58a: Deleting consensus metadata
I20260929 02:46:02.531638 1499 catalog_manager.cc:4841] TS 856cc9e75d03495eb5355b04b6209630 (127.0.97.1:44427): tablet 7a4cedb3fdf84f079dfdad5a9d08070c (table test-table [id=8efb21e67bca4a3ca67de3e59605248e]) successfully deleted
I20260929 02:46:02.531833 1496 catalog_manager.cc:4841] TS c2350a618774408b90fc31f21dddb58a (127.0.97.2:41387): tablet 7a4cedb3fdf84f079dfdad5a9d08070c (table test-table [id=8efb21e67bca4a3ca67de3e59605248e]) successfully deleted
I20260929 02:46:02.534955 2091 ts_tablet_manager.cc:1918] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:02.535104 2091 log.cc:1196] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/7a4cedb3fdf84f079dfdad5a9d08070c
I20260929 02:46:02.535567 2091 ts_tablet_manager.cc:1939] T 7a4cedb3fdf84f079dfdad5a9d08070c P 81600b06281d4002ad3a6bd3f52e52e6: Deleting consensus metadata
I20260929 02:46:02.536693 1497 catalog_manager.cc:4841] TS 81600b06281d4002ad3a6bd3f52e52e6 (127.0.97.3:36333): tablet 7a4cedb3fdf84f079dfdad5a9d08070c (table test-table [id=8efb21e67bca4a3ca67de3e59605248e]) successfully deleted
Sep 29 02:46:02 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:02.547642 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.509902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.97.1:44427)
0929 02:46:02.510075 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.510082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.521210 (+ 11128us) server_negotiation.cc:407] Connection header received
0929 02:46:02.521515 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.521521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.521613 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.521688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:02.525326 (+ 3638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.526511 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.532342 (+ 5831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.533096 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.533208 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.539626 (+ 6418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:02.539645 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:02.539666 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:02.539728 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:02.544073 (+ 4345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.545108 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.545119 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.545123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.545289 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.545739 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.545744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.545748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.546249 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:02.546582 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.546976 (+ 394us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.547257 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 02:46:02.548916 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59276
I20260929 02:46:02.589691 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.563366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.0.97.62:39735)
0929 02:46:02.563603 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.563610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.570919 (+ 7309us) server_negotiation.cc:407] Connection header received
0929 02:46:02.571219 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.571226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.571319 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.571418 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:02.573636 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.574888 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.576988 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.577700 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.577810 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.581337 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:02.581345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:02.581351 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:02.581396 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:02.585693 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.586634 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.586641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.586645 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.586795 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.587246 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.587252 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.587256 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.587742 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:02.588012 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.589005 (+ 993us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.589294 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260929 02:46:02.591172 1511 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55558
I20260929 02:46:02.593714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1583
I20260929 02:46:02.613260 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.594864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.0.97.1:44427)
0929 02:46:02.595028 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.595034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.596281 (+ 1247us) server_negotiation.cc:407] Connection header received
0929 02:46:02.596446 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.596452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.596597 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.596706 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:02.598279 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.599445 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.601419 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.602176 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.602307 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.605460 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:02.605467 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:02.605472 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:02.605513 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:02.609827 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.610696 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.610702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.610707 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.610916 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.611371 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.611376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.611380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.611749 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:02.611977 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.612588 (+ 611us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.612866 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 02:46:02.614734 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59282
W20260929 02:46:02.615391 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:02.615592 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 1583
I20260929 02:46:02.635469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1723
Sep 29 02:46:02 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:46:02.652889 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.636256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.0.97.2:41387)
0929 02:46:02.636434 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.636442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.636465 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:46:02.636727 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.636732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.636816 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.636930 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:02.638008 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.638795 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.640186 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.640687 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.640823 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.645536 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:02.645555 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:02.645577 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:02.645625 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:02.649577 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.650389 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.650399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.650403 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.650568 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.651110 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.651117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.651121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.651566 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:02.651903 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.652201 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.652442 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 02:46:02.653707 1853 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46958
W20260929 02:46:02.654428 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:02.654644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 1723
I20260929 02:46:02.683321 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1862
Sep 29 02:46:02 dist-test-slave-6tjh krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649961, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:46:02.701515 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:02.684204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51866 (local address 127.0.97.3:36333)
0929 02:46:02.684408 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:02.684415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:02.684439 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:46:02.684572 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:02.684578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:02.684687 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:02.684825 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:02.686107 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.687243 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.688737 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:02.689449 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:02.689615 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:02.694220 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:02.694234 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:02.694255 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:02.694304 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:02.698569 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.699236 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.699248 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.699254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.699419 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:02.699784 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:02.699789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:02.699793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:02.700323 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:02.700661 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:02.700904 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:46:02.701130 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 02:46:02.702309 1992 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51866
W20260929 02:46:02.703009 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1862: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:02.703163 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 1862
I20260929 02:46:02.723112 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 1480
I20260929 02:46:02.724323 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:02.840727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 1480
2026-09-29T02:46:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5136 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:03 dist-test-slave-6tjh krb5kdc[2100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:03 dist-test-slave-6tjh krb5kdc[2100](info): set up 2 sockets
Sep 29 02:46:03 dist-test-slave-6tjh krb5kdc[2100](info): commencing operation
krb5kdc: starting...
W20260929 02:46:04.020794 388 mini_kdc.cc:121] Time spent starting KDC: real 1.135s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:04 dist-test-slave-6tjh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:04.208787 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36921
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:04.528775 2113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:04.529068 2113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:04.529234 2113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:04.536353 2113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:04.536449 2113 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:46:04.536530 2113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:04.536651 2113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:04.536767 2113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:04.536882 2113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:04.546516 2113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36921
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:04.547830 2113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:04.549228 2113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:04.561334 2118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:04.561275 2119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:04.561625 2121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:04.562098 2113 server_base.cc:1034] running on GCE node
I20260929 02:46:04.562917 2113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:04.564352 2113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:04.565616 2113 hybrid_clock.cc:648] HybridClock initialized: now 1790649964565539 us; error 51 us; skew 500 ppm
I20260929 02:46:04.566015 2113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:04 dist-test-slave-6tjh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:04.569489 2113 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:04.571091 2113 webserver.cc:460] Webserver started at http://127.0.97.62:46863/ using document root <none> and password file <none>
I20260929 02:46:04.571820 2113 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:04.571954 2113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:04.572263 2113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:04.575237 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "c325640a5e594d6280f5b29f7aee512e"
format_stamp: "Formatted at 2026-09-29 02:46:04 on dist-test-slave-6tjh"
server_key: "ef4a3f9dc8e8598eb6115269448b5d16"
server_key_iv: "6c4be549eab54867557bfdd8dc84f74f"
server_key_version: "encryptionkey@0"
I20260929 02:46:04.575996 2113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "c325640a5e594d6280f5b29f7aee512e"
format_stamp: "Formatted at 2026-09-29 02:46:04 on dist-test-slave-6tjh"
server_key: "ef4a3f9dc8e8598eb6115269448b5d16"
server_key_iv: "6c4be549eab54867557bfdd8dc84f74f"
server_key_version: "encryptionkey@0"
I20260929 02:46:04.581146 2113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:46:04.584700 2128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:04.586138 2113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:04.586337 2113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "c325640a5e594d6280f5b29f7aee512e"
format_stamp: "Formatted at 2026-09-29 02:46:04 on dist-test-slave-6tjh"
server_key: "ef4a3f9dc8e8598eb6115269448b5d16"
server_key_iv: "6c4be549eab54867557bfdd8dc84f74f"
server_key_version: "encryptionkey@0"
I20260929 02:46:04.586557 2113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:04.615542 2113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:04.620015 2113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:04.620296 2113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:04.643872 2113 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36921
I20260929 02:46:04.643893 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36921 every 8 connection(s)
I20260929 02:46:04.645375 2113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:04.646520 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 2113
I20260929 02:46:04.646759 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:04.647460 388 external_mini_cluster.cc:1428] Setting key c56015b7e2c273a49c3b78436ea1773c
I20260929 02:46:04.649757 2190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:46:04 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:04.663333 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Bootstrap starting.
I20260929 02:46:04.666579 2190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:04.667626 2190 log.cc:824] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:04.670604 2190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: No bootstrap required, opened a new log
I20260929 02:46:04.670815 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:04.649583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58008 (local address 127.0.97.62:36921)
0929 02:46:04.650437 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:04.650449 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:04.650492 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:46:04.652304 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:04.652328 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:04.652834 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:04.653210 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:04.654979 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:04.656360 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:04.658109 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:04.658839 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:04.658982 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:04.663656 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:04.663665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:04.663681 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:04.663721 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:04.666823 (+ 3102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:04.667520 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:04.667527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:04.667534 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:04.667695 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:04.668072 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:04.668080 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:04.668083 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:04.668680 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:04.668982 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:04.669435 (+ 453us) server_negotiation.cc:299] Negotiation successful
0929 02:46:04.669836 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":428,"threads_started":1}
I20260929 02:46:04.678396 2190 raft_consensus.cc:348] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325640a5e594d6280f5b29f7aee512e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36921 } }
I20260929 02:46:04.678761 2190 raft_consensus.cc:374] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:04.678874 2190 raft_consensus.cc:729] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c325640a5e594d6280f5b29f7aee512e, State: Initialized, Role: FOLLOWER
I20260929 02:46:04.679478 2190 consensus_queue.cc:260] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325640a5e594d6280f5b29f7aee512e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36921 } }
I20260929 02:46:04.679706 2190 raft_consensus.cc:388] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:04.679847 2190 raft_consensus.cc:482] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:04.680024 2190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:04.682485 2190 raft_consensus.cc:504] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325640a5e594d6280f5b29f7aee512e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36921 } }
I20260929 02:46:04.682969 2190 leader_election.cc:302] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c325640a5e594d6280f5b29f7aee512e; no voters:
I20260929 02:46:04.683555 2190 leader_election.cc:288] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:04.683682 2195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:04.684455 2195 raft_consensus.cc:686] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [term 1 LEADER]: Becoming Leader. State: Replica: c325640a5e594d6280f5b29f7aee512e, State: Running, Role: LEADER
I20260929 02:46:04.685000 2195 consensus_queue.cc:237] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325640a5e594d6280f5b29f7aee512e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36921 } }
I20260929 02:46:04.685725 2190 sys_catalog.cc:564] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:04.688010 2197 sys_catalog.cc:455] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c325640a5e594d6280f5b29f7aee512e. Latest consensus state: current_term: 1 leader_uuid: "c325640a5e594d6280f5b29f7aee512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325640a5e594d6280f5b29f7aee512e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36921 } } }
I20260929 02:46:04.688274 2196 sys_catalog.cc:455] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c325640a5e594d6280f5b29f7aee512e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325640a5e594d6280f5b29f7aee512e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36921 } } }
I20260929 02:46:04.688498 2197 sys_catalog.cc:458] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:04.688519 2196 sys_catalog.cc:458] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:04.696611 2204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:04.701869 2204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:04.710615 2204 catalog_manager.cc:1347] Generated new cluster ID: 2d0d545de5554c0f90260879cbfbeb75
I20260929 02:46:04.710745 2204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:04.730660 2204 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:04.732671 2204 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:04.741649 2204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Generated new TSK 0
I20260929 02:46:04.742741 2204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:46:04.800750 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36921
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:05.125774 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:05.126089 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:05.126266 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:05.133517 2216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:05.133665 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:05.133824 2216 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:46:05.133912 2216 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:05.134053 2216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:05.143850 2216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:05.145229 2216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:05.146569 2216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:05.159495 2221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:05.159521 2222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:05.159554 2224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:05.160302 2216 server_base.cc:1034] running on GCE node
I20260929 02:46:05.160934 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:05.161882 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:05.163079 2216 hybrid_clock.cc:648] HybridClock initialized: now 1790649965163037 us; error 34 us; skew 500 ppm
I20260929 02:46:05.163383 2216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:05 dist-test-slave-6tjh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:05.166487 2216 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:46:05.167855 2216 webserver.cc:460] Webserver started at http://127.0.97.1:34493/ using document root <none> and password file <none>
I20260929 02:46:05.168586 2216 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:05.168702 2216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:05.168949 2216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:05.171881 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "763f17acab1941d792bfa92beeb2162d"
format_stamp: "Formatted at 2026-09-29 02:46:05 on dist-test-slave-6tjh"
server_key: "e9de533ec5fbdd69fe3a6b24371d8ae9"
server_key_iv: "b27e7dc721dfb1d1e33114ac6766cf19"
server_key_version: "encryptionkey@0"
I20260929 02:46:05.172813 2216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "763f17acab1941d792bfa92beeb2162d"
format_stamp: "Formatted at 2026-09-29 02:46:05 on dist-test-slave-6tjh"
server_key: "e9de533ec5fbdd69fe3a6b24371d8ae9"
server_key_iv: "b27e7dc721dfb1d1e33114ac6766cf19"
server_key_version: "encryptionkey@0"
I20260929 02:46:05.177704 2216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:05.181159 2231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:05.182492 2216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:46:05.182615 2216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "763f17acab1941d792bfa92beeb2162d"
format_stamp: "Formatted at 2026-09-29 02:46:05 on dist-test-slave-6tjh"
server_key: "e9de533ec5fbdd69fe3a6b24371d8ae9"
server_key_iv: "b27e7dc721dfb1d1e33114ac6766cf19"
server_key_version: "encryptionkey@0"
I20260929 02:46:05.182776 2216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:05.203487 2216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:05.207473 2216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:05.207712 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:05.210376 2216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:05.210510 2216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:05.210635 2216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:05.210744 2216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:46:05 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:05.236408 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.210747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57309 (local address 127.0.97.62:36921)
0929 02:46:05.211621 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:05.211627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:05.212524 (+ 897us) server_negotiation.cc:407] Connection header received
0929 02:46:05.214573 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:05.214582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:05.214702 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:05.214837 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:05.217144 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.218424 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.221330 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.222091 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.222273 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.228342 (+ 6069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:05.228352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:05.228356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:05.228407 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:05.231778 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.233036 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.233042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.233046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.233206 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.233880 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.233886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.233892 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.234323 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:05.234503 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:05.235675 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0929 02:46:05.235946 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":629,"threads_started":1}
I20260929 02:46:05.238478 2241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.211463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36921 (local address 127.0.97.1:57309)
0929 02:46:05.212346 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:05.212394 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:05.214316 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:05.215092 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:05.215106 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:05.215850 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:05.216919 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.216951 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.218625 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.218634 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.221138 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.221159 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.222412 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.222422 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.222803 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.223831 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:05.223874 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:05.227986 (+ 4112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:05.231966 (+ 3980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.231981 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.232002 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.232864 (+ 862us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.233339 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.233347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.233353 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.233719 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.234490 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:05.234501 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:05.235118 (+ 617us) client_negotiation.cc:769] Sending connection context
0929 02:46:05.235552 (+ 434us) client_negotiation.cc:240] Negotiation successful
0929 02:46:05.236344 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":382,"threads_started":1}
I20260929 02:46:05.253221 2216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42441
I20260929 02:46:05.253243 2351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42441 every 8 connection(s)
I20260929 02:46:05.254784 2216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:05.256999 2352 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36921
I20260929 02:46:05.257244 2352 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:05.257910 2352 heartbeater.cc:511] Master 127.0.97.62:36921 requested a full tablet report, sending...
I20260929 02:46:05.260059 2145 ts_manager.cc:194] Registered new tserver with Master: 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441)
I20260929 02:46:05.260510 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 2216
I20260929 02:46:05.260741 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:05.261502 388 external_mini_cluster.cc:1428] Setting key c3f47914efd1f743d410410e1d37a0c3
I20260929 02:46:05.261924 2145 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57309
I20260929 02:46:05.263347 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.246028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36615 (local address 127.0.97.62:36921)
0929 02:46:05.246194 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:05.246200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:05.249139 (+ 2939us) server_negotiation.cc:407] Connection header received
0929 02:46:05.249338 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:05.249358 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:05.249450 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:05.249553 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:05.251243 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.252325 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.254389 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.254803 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.254952 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.257440 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:05.257480 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:05.257483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:05.257513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:05.260348 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.261139 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.261145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.261148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.261322 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.261704 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.261709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.261712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.262105 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:05.262254 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:05.262693 (+ 439us) server_negotiation.cc:299] Negotiation successful
0929 02:46:05.263032 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:46:05.263350 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.245737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36921 (local address 127.0.97.1:36615)
0929 02:46:05.249044 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:05.249059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:05.249209 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:05.249624 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:05.249632 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:05.250431 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:05.251008 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.251024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.252518 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.252524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.254256 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.254269 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.255017 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.255025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.255232 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.256142 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:05.256177 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:05.257265 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:05.260485 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.260490 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.260494 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.261000 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.261443 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.261448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.261452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.261592 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.262229 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:05.262234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:05.262417 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 02:46:05.262563 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 02:46:05.262866 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3178,"thread_start_us":2939,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:05.302028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36921
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:05.622567 2356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:05.622874 2356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:05.622977 2356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:05.630151 2356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:05.630259 2356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:05.630344 2356 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:46:05.630398 2356 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:05.630460 2356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:05.639997 2356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:05.641350 2356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:05.642668 2356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:05.656201 2361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:05.656220 2362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:05.656957 2364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:05.662150 2356 server_base.cc:1034] running on GCE node
I20260929 02:46:05.662824 2356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:05.663921 2356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:05.665117 2356 hybrid_clock.cc:648] HybridClock initialized: now 1790649965665068 us; error 44 us; skew 500 ppm
I20260929 02:46:05.665454 2356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:05 dist-test-slave-6tjh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:05.674059 2356 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:05.675664 2356 webserver.cc:460] Webserver started at http://127.0.97.2:33153/ using document root <none> and password file <none>
I20260929 02:46:05.676430 2356 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:05.676585 2356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:05.676904 2356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:05.680022 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "1f003bde72b74706a5fffad865915b35"
format_stamp: "Formatted at 2026-09-29 02:46:05 on dist-test-slave-6tjh"
server_key: "d10e7dbec4ac7bb59eaaf846d55c741f"
server_key_iv: "258b1dfef9475a1320ddced5a1f430bf"
server_key_version: "encryptionkey@0"
I20260929 02:46:05.681053 2356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1f003bde72b74706a5fffad865915b35"
format_stamp: "Formatted at 2026-09-29 02:46:05 on dist-test-slave-6tjh"
server_key: "d10e7dbec4ac7bb59eaaf846d55c741f"
server_key_iv: "258b1dfef9475a1320ddced5a1f430bf"
server_key_version: "encryptionkey@0"
I20260929 02:46:05.686550 2356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:05.690409 2371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:05.691823 2356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:46:05.692006 2356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "1f003bde72b74706a5fffad865915b35"
format_stamp: "Formatted at 2026-09-29 02:46:05 on dist-test-slave-6tjh"
server_key: "d10e7dbec4ac7bb59eaaf846d55c741f"
server_key_iv: "258b1dfef9475a1320ddced5a1f430bf"
server_key_version: "encryptionkey@0"
I20260929 02:46:05.692220 2356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:05.710367 2356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:05.714366 2356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:05.714655 2356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:05.717825 2356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:05.717960 2356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:05.718089 2356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:05.718160 2356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:46:05 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649965, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:05.749208 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.718266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57457 (local address 127.0.97.62:36921)
0929 02:46:05.718468 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:05.718476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:05.719978 (+ 1502us) server_negotiation.cc:407] Connection header received
0929 02:46:05.722375 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:05.722393 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:05.722490 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:05.722602 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:05.725049 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.726164 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.728760 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.729530 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.729647 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.741114 (+ 11467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:05.741125 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:05.741130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:05.741178 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:05.744795 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.745939 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.745944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.745946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.746073 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.746668 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.746675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.746679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.747131 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:05.747247 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:05.748471 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0929 02:46:05.748775 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:46:05.751180 2380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.718921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36921 (local address 127.0.97.2:57457)
0929 02:46:05.719802 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:05.719847 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:05.722086 (+ 2239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:05.722862 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:05.722871 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:05.723723 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:05.724786 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.724821 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.726413 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.726418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.728530 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.728567 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.730101 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.730109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.730461 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.736939 (+ 6478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:05.736984 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:05.740751 (+ 3767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:05.744997 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.745013 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.745046 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.745750 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.746173 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.746184 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.746190 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.746505 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.747275 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:05.747290 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:05.747932 (+ 642us) client_negotiation.cc:769] Sending connection context
0929 02:46:05.748341 (+ 409us) client_negotiation.cc:240] Negotiation successful
0929 02:46:05.749126 (+ 785us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":374,"threads_started":1}
I20260929 02:46:05.775437 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.760857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36217 (local address 127.0.97.62:36921)
0929 02:46:05.761012 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:05.761018 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:05.761157 (+ 139us) server_negotiation.cc:407] Connection header received
0929 02:46:05.761342 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:05.761351 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:05.761448 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:05.761578 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:05.762877 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.763973 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.766091 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.766551 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.766681 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.769113 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:05.769118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:05.769121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:05.769150 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:05.772409 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.773360 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.773368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.773374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.773532 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:05.773959 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:05.773964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:05.773968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:05.774282 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:05.774432 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:05.774866 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 02:46:05.775076 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:46:05.775544 2450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:05.760259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36921 (local address 127.0.97.2:36217)
0929 02:46:05.761011 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:05.761033 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:05.761190 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:05.761629 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:05.761636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:05.762232 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:05.762726 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.762737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.764154 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.764161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.765884 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:05.765908 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:05.766749 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:05.766761 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:05.766955 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:05.767598 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:05.767614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:05.768889 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:05.772537 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.772569 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.772574 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.773230 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.773675 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:05.773681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:05.773685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:05.773822 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:05.774400 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:05.774405 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:05.774593 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 02:46:05.774769 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 02:46:05.775108 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":749,"threads_started":1}
I20260929 02:46:05.776775 2356 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39825
I20260929 02:46:05.776785 2490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39825 every 8 connection(s)
I20260929 02:46:05.778373 2356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:05.780587 2491 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36921
I20260929 02:46:05.780786 2491 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:05.781347 2491 heartbeater.cc:511] Master 127.0.97.62:36921 requested a full tablet report, sending...
I20260929 02:46:05.782495 2144 ts_manager.cc:194] Registered new tserver with Master: 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825)
I20260929 02:46:05.783345 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57457
I20260929 02:46:05.787752 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 2356
I20260929 02:46:05.787922 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:05.788344 388 external_mini_cluster.cc:1428] Setting key fb245794ee86519fb480d26cff765e35
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:05.828444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36921
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:06.160861 2495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:06.161201 2495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:06.161382 2495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:06.168845 2495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:06.169032 2495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:06.169175 2495 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 02:46:06.169260 2495 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:06.169399 2495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:06.179359 2495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36921
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:06.180738 2495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:06.182081 2495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:06.195891 2501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:06.195861 2503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:06.196887 2500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:06.201695 2495 server_base.cc:1034] running on GCE node
I20260929 02:46:06.202396 2495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:06.203469 2495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:06.204717 2495 hybrid_clock.cc:648] HybridClock initialized: now 1790649966204672 us; error 57 us; skew 500 ppm
I20260929 02:46:06.205194 2495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:06 dist-test-slave-6tjh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:06.208735 2495 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:06.210275 2495 webserver.cc:460] Webserver started at http://127.0.97.3:38675/ using document root <none> and password file <none>
I20260929 02:46:06.211015 2495 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:06.211163 2495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:06.211468 2495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:06.214509 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "a4edb27278484b4cb4e7945424cfaf37"
format_stamp: "Formatted at 2026-09-29 02:46:06 on dist-test-slave-6tjh"
server_key: "c87a6476407c1c06868b46ee1ce473b9"
server_key_iv: "afc8112100b5d6cd0048dd60a0f67d65"
server_key_version: "encryptionkey@0"
I20260929 02:46:06.215315 2495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a4edb27278484b4cb4e7945424cfaf37"
format_stamp: "Formatted at 2026-09-29 02:46:06 on dist-test-slave-6tjh"
server_key: "c87a6476407c1c06868b46ee1ce473b9"
server_key_iv: "afc8112100b5d6cd0048dd60a0f67d65"
server_key_version: "encryptionkey@0"
I20260929 02:46:06.220356 2495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:46:06.223752 2510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:06.225208 2495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:06.225389 2495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "a4edb27278484b4cb4e7945424cfaf37"
format_stamp: "Formatted at 2026-09-29 02:46:06 on dist-test-slave-6tjh"
server_key: "c87a6476407c1c06868b46ee1ce473b9"
server_key_iv: "afc8112100b5d6cd0048dd60a0f67d65"
server_key_version: "encryptionkey@0"
I20260929 02:46:06.225595 2495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:06.249857 2495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:06.254558 2495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:06.254871 2495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:06.257786 2495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:06.257938 2495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:06.258056 2495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:06.258141 2495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:06.265712 2352 heartbeater.cc:503] Master 127.0.97.62:36921 was elected leader, sending a full tablet report...
Sep 29 02:46:06 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:06.284106 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.258550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42529 (local address 127.0.97.62:36921)
0929 02:46:06.258765 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.258770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.260475 (+ 1705us) server_negotiation.cc:407] Connection header received
0929 02:46:06.263118 (+ 2643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.263127 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.263242 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.263387 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:06.266211 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.267676 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.270217 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.270682 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.270829 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.276253 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:06.276258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:06.276261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:06.276315 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:06.279939 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.281041 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.281045 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.281047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.281245 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.281934 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.281941 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.281947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.282318 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:06.282494 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.283596 (+ 1102us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.283827 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:46:06.286229 2519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.259336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36921 (local address 127.0.97.3:42529)
0929 02:46:06.260274 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.260319 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.262663 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.263742 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.263756 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.264748 (+ 992us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:06.265820 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.265851 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.267867 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.267873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.270032 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.270044 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.270928 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.270936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.271175 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.272011 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:06.272053 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:06.275969 (+ 3916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:06.280170 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:06.280182 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:06.280204 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:06.280878 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:06.281388 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:06.281395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:06.281402 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:06.281770 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:06.282476 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:06.282493 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:06.283070 (+ 577us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.283464 (+ 394us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.284180 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":426,"threads_started":1}
I20260929 02:46:06.319936 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.295848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49063 (local address 127.0.97.62:36921)
0929 02:46:06.296076 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.296085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.303151 (+ 7066us) server_negotiation.cc:407] Connection header received
0929 02:46:06.303460 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.303469 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.303564 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.303662 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:06.305442 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.306284 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.308907 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.309376 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.309552 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.312765 (+ 3213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:06.312770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:06.312773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:06.312807 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:06.316664 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.317826 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.317830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.317833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.317978 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.318440 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.318444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.318446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.318742 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:06.318865 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.319486 (+ 621us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.319675 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:46:06.320528 2591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.295841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36921 (local address 127.0.97.3:49063)
0929 02:46:06.302992 (+ 7151us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.303017 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.303289 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.303729 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.303737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.304581 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:06.305225 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.305244 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.306469 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.306481 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.308742 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.308758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.309852 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.309859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.310033 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.311065 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:06.311099 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:06.312524 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:06.316875 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:06.316884 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:06.316892 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:06.317659 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:06.318097 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:06.318105 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:06.318112 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:06.318307 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:06.318861 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:06.318868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:06.319140 (+ 272us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.319335 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.319814 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6948,"thread_start_us":6762,"threads_started":1}
I20260929 02:46:06.322779 2495 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41519
I20260929 02:46:06.322814 2629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41519 every 8 connection(s)
I20260929 02:46:06.324105 2495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:06.326246 2630 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36921
I20260929 02:46:06.326530 2630 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:06.327234 2630 heartbeater.cc:511] Master 127.0.97.62:36921 requested a full tablet report, sending...
I20260929 02:46:06.327548 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 2495
I20260929 02:46:06.327718 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:06.328295 388 external_mini_cluster.cc:1428] Setting key e2504e5c6a56362caca16cc436ce5993
I20260929 02:46:06.328814 2144 ts_manager.cc:194] Registered new tserver with Master: a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:06.330030 2144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42529
I20260929 02:46:06.331334 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:46:06 dist-test-slave-6tjh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:46:06 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:06.378183 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.364238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.0.97.62:36921)
0929 02:46:06.364471 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.364477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.364692 (+ 215us) server_negotiation.cc:407] Connection header received
0929 02:46:06.364919 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.364931 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.364994 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.365101 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:06.366470 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.367196 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.368444 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.368938 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.369083 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.372952 (+ 3869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:06.372956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:06.372959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:06.372992 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:06.375500 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.376198 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.376210 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.376212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.376331 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.376803 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.376807 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.376809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.377055 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:06.377305 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.377611 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.377839 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:46:06.382570 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:06.390050 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:06.426990 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.402337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42441 (local address 127.0.0.1:47934)
0929 02:46:06.414149 (+ 11812us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.414208 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.414412 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.419117 (+ 4705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.419127 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.419158 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:06.419636 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.419652 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.421584 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.421591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.423701 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.423715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.425877 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.425886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.426108 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.426187 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.426321 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.426510 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11467,"mutex_wait_us":3,"thread_start_us":12295,"threads_started":1}
I20260929 02:46:06.427953 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.402430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.0.97.1:42441)
0929 02:46:06.418023 (+ 15593us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.418035 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.418069 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:46:06.418186 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.418195 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.418483 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.418718 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:06.419861 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.421436 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.423894 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.425408 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.425529 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.425721 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.427502 (+ 1781us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.427679 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15424,"thread_start_us":15309,"threads_started":1}
I20260929 02:46:06.428761 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.415644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39825 (local address 127.0.0.1:59136)
0929 02:46:06.418268 (+ 2624us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.418285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.418390 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.421567 (+ 3177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.421576 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.421626 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:06.422390 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.422408 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.424159 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.424168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.425876 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.425894 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.427610 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.427617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.427803 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.427850 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.427959 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.428305 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2523,"mutex_wait_us":28,"thread_start_us":3239,"threads_started":1}
I20260929 02:46:06.430946 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.415986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59136 (local address 127.0.97.2:39825)
0929 02:46:06.420970 (+ 4984us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.420979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.421007 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:46:06.421090 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.421097 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.421327 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.421546 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:06.422623 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.423982 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.426015 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.427397 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.427522 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.427723 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.428027 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.430696 (+ 2669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4823,"thread_start_us":4715,"threads_started":1}
I20260929 02:46:06.431303 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.403499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.0.97.3:41519)
0929 02:46:06.423060 (+ 19561us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.423067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.423104 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:46:06.423180 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.423185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.423359 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.423565 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:06.424238 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.425823 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.428487 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.430157 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.430294 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.430506 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.430748 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.430898 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19409,"thread_start_us":20338,"threads_started":1}
I20260929 02:46:06.431571 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.402937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41519 (local address 127.0.0.1:37170)
0929 02:46:06.414230 (+ 11293us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.414256 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.414417 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.423569 (+ 9152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.423574 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.423615 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:06.424033 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.424047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.425959 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.425967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.428324 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.428338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.430383 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.430390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.430527 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.430549 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.430710 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.430838 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11093,"mutex_wait_us":1,"thread_start_us":11695,"threads_started":1}
I20260929 02:46:06.432047 2282 tablet_service.cc:1467] Processing CreateTablet for tablet b519126285964517985b244b217ac1e0 (DEFAULT_TABLE table=test-table [id=bc9f5959e27741b783d382134d9615c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:06.434410 2560 tablet_service.cc:1467] Processing CreateTablet for tablet b519126285964517985b244b217ac1e0 (DEFAULT_TABLE table=test-table [id=bc9f5959e27741b783d382134d9615c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:06.434477 2421 tablet_service.cc:1467] Processing CreateTablet for tablet b519126285964517985b244b217ac1e0 (DEFAULT_TABLE table=test-table [id=bc9f5959e27741b783d382134d9615c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:06.435570 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b519126285964517985b244b217ac1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.435657 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b519126285964517985b244b217ac1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.435662 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b519126285964517985b244b217ac1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.446480 2649 tablet_bootstrap.cc:492] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Bootstrap starting.
I20260929 02:46:06.449747 2651 tablet_bootstrap.cc:492] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Bootstrap starting.
I20260929 02:46:06.450047 2649 tablet_bootstrap.cc:654] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.451489 2649 log.cc:824] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:06.453307 2651 tablet_bootstrap.cc:654] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.453610 2650 tablet_bootstrap.cc:492] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Bootstrap starting.
I20260929 02:46:06.454600 2651 log.cc:824] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:06.455473 2649 tablet_bootstrap.cc:492] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: No bootstrap required, opened a new log
I20260929 02:46:06.455762 2649 ts_tablet_manager.cc:1397] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 02:46:06.456459 2650 tablet_bootstrap.cc:654] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.457419 2650 log.cc:824] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:06.457660 2651 tablet_bootstrap.cc:492] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: No bootstrap required, opened a new log
I20260929 02:46:06.457921 2651 ts_tablet_manager.cc:1397] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:46:06.459776 2650 tablet_bootstrap.cc:492] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: No bootstrap required, opened a new log
I20260929 02:46:06.460060 2650 ts_tablet_manager.cc:1397] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260929 02:46:06.466566 2649 raft_consensus.cc:348] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.467043 2649 raft_consensus.cc:374] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.467134 2649 raft_consensus.cc:729] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f003bde72b74706a5fffad865915b35, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.467746 2650 raft_consensus.cc:348] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.467753 2649 consensus_queue.cc:260] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.468072 2650 raft_consensus.cc:374] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.468156 2650 raft_consensus.cc:729] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.468928 2491 heartbeater.cc:503] Master 127.0.97.62:36921 was elected leader, sending a full tablet report...
I20260929 02:46:06.468839 2650 consensus_queue.cc:260] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.469637 2649 ts_tablet_manager.cc:1428] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260929 02:46:06.468526 2651 raft_consensus.cc:348] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.470180 2651 raft_consensus.cc:374] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.470287 2651 raft_consensus.cc:729] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4edb27278484b4cb4e7945424cfaf37, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.470381 2650 ts_tablet_manager.cc:1428] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260929 02:46:06.470932 2651 consensus_queue.cc:260] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.471979 2630 heartbeater.cc:503] Master 127.0.97.62:36921 was elected leader, sending a full tablet report...
I20260929 02:46:06.472600 2651 ts_tablet_manager.cc:1428] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Time spent starting tablet: real 0.015s user 0.009s sys 0.004s
W20260929 02:46:06.507266 2353 tablet.cc:2369] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:06.514166 2656 raft_consensus.cc:482] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:06.514480 2656 raft_consensus.cc:504] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.516187 2656 leader_election.cc:288] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:06.523257 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.516761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34979 (local address 127.0.97.3:41519)
0929 02:46:06.516995 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.517004 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.517775 (+ 771us) server_negotiation.cc:407] Connection header received
0929 02:46:06.518018 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.518023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.518078 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.518185 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:06.518849 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.519504 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.521513 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.522276 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.522405 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.522533 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.522866 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.522990 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:46:06.523288 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.516615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49599 (local address 127.0.97.2:39825)
0929 02:46:06.516847 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.516855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.517816 (+ 961us) server_negotiation.cc:407] Connection header received
0929 02:46:06.517988 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.518001 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.518060 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.518129 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:06.518791 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.519523 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.521519 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.522278 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.522446 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.522556 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.522917 (+ 361us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.523040 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:46:06.523492 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.516579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41519 (local address 127.0.97.1:34979)
0929 02:46:06.517647 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.517667 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.517819 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.518214 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.518218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.518240 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:06.518672 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.518681 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.519640 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.519646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.521352 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.521364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.522505 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.522512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.522696 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.522735 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.522809 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.522966 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":95,"thread_start_us":700,"threads_started":1}
I20260929 02:46:06.523831 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.516597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39825 (local address 127.0.97.1:49599)
0929 02:46:06.517666 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.517689 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.517848 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.518187 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.518191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.518222 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:06.518625 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.518634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.519685 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.519688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.521359 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.521371 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.522546 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.522554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.522735 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.522757 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.522864 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.522987 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":65,"thread_start_us":819,"threads_started":1}
I20260929 02:46:06.524493 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b519126285964517985b244b217ac1e0" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37" is_pre_election: true
I20260929 02:46:06.524875 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b519126285964517985b244b217ac1e0" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f003bde72b74706a5fffad865915b35" is_pre_election: true
I20260929 02:46:06.525053 2580 raft_consensus.cc:2454] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 0.
I20260929 02:46:06.525370 2445 raft_consensus.cc:2454] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 0.
I20260929 02:46:06.525768 2233 leader_election.cc:302] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 763f17acab1941d792bfa92beeb2162d, a4edb27278484b4cb4e7945424cfaf37; no voters:
I20260929 02:46:06.526248 2656 raft_consensus.cc:2781] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:06.526401 2656 raft_consensus.cc:482] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:06.526502 2656 raft_consensus.cc:3037] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:06.528764 2656 raft_consensus.cc:504] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.529335 2656 leader_election.cc:288] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 election: Requested vote from peers 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:06.529843 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b519126285964517985b244b217ac1e0" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f003bde72b74706a5fffad865915b35"
I20260929 02:46:06.529843 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b519126285964517985b244b217ac1e0" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37"
I20260929 02:46:06.530052 2580 raft_consensus.cc:3037] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:06.530043 2445 raft_consensus.cc:3037] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:46:06.530566 2492 tablet.cc:2369] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:06.533275 2580 raft_consensus.cc:2454] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 1.
I20260929 02:46:06.533275 2445 raft_consensus.cc:2454] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 1.
I20260929 02:46:06.533699 2235 leader_election.cc:302] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f003bde72b74706a5fffad865915b35, 763f17acab1941d792bfa92beeb2162d; no voters:
I20260929 02:46:06.534039 2656 raft_consensus.cc:2781] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:06.534451 2656 raft_consensus.cc:686] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 1 LEADER]: Becoming Leader. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Running, Role: LEADER
I20260929 02:46:06.534900 2656 consensus_queue.cc:237] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.539430 2145 catalog_manager.cc:5495] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d reported cstate change: term changed from 0 to 1, leader changed from <none> to 763f17acab1941d792bfa92beeb2162d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } health_report { overall_health: UNKNOWN } } }
W20260929 02:46:06.575877 2631 tablet.cc:2369] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 02:46:06 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649964, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:06.597597 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.583265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36921 (local address 127.0.0.1:58014)
0929 02:46:06.583833 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.583849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.583974 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.584275 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.584278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.585158 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:06.585439 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.585448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.586553 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.586557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.587634 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.587644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.588418 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.588423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.588586 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.589255 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:06.589272 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:06.591761 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:06.594772 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:06.594782 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:06.594788 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:06.595322 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:06.595716 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:06.595723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:06.595726 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:06.595894 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:06.596437 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:06.596445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:06.596677 (+ 232us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.596866 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.597146 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":309,"threads_started":1}
I20260929 02:46:06.597627 2239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.583213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.0.97.62:36921)
0929 02:46:06.583425 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.583429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.583934 (+ 505us) server_negotiation.cc:407] Connection header received
0929 02:46:06.584069 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.584073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.584131 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.584230 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:06.585570 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.586420 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.587795 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.588221 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.588340 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.591931 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:06.591935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:06.591938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:06.591966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:06.594626 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.595437 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.595441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.595444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.595563 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:06.596009 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:06.596013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:06.596015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:06.596290 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:06.596408 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.596960 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.597277 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":67}
I20260929 02:46:06.602732 2145 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58014:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:46:06.603730 2145 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:06.610291 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 90a56464fc104fe3b986f6a865a04500 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0e143feb54343708a3af0d0b3fced20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:06.610219 2282 tablet_service.cc:1467] Processing CreateTablet for tablet 90a56464fc104fe3b986f6a865a04500 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0e143feb54343708a3af0d0b3fced20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:06.610497 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 90a56464fc104fe3b986f6a865a04500 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b0e143feb54343708a3af0d0b3fced20]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:06.611191 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a56464fc104fe3b986f6a865a04500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.611191 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a56464fc104fe3b986f6a865a04500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.611382 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90a56464fc104fe3b986f6a865a04500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.616230 2651 tablet_bootstrap.cc:492] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: Bootstrap starting.
I20260929 02:46:06.617515 2651 tablet_bootstrap.cc:654] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.618084 2649 tablet_bootstrap.cc:492] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: Bootstrap starting.
I20260929 02:46:06.618148 2650 tablet_bootstrap.cc:492] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: Bootstrap starting.
I20260929 02:46:06.619663 2651 tablet_bootstrap.cc:492] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: No bootstrap required, opened a new log
I20260929 02:46:06.619768 2649 tablet_bootstrap.cc:654] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.619774 2650 tablet_bootstrap.cc:654] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.619812 2651 ts_tablet_manager.cc:1397] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:46:06.620661 2651 raft_consensus.cc:348] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.620961 2651 raft_consensus.cc:374] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.621065 2651 raft_consensus.cc:729] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4edb27278484b4cb4e7945424cfaf37, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.621353 2651 consensus_queue.cc:260] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.622012 2651 ts_tablet_manager.cc:1428] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:46:06.621846 2657 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.622110 2650 tablet_bootstrap.cc:492] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: No bootstrap required, opened a new log
I20260929 02:46:06.622195 2657 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:06.622258 2650 ts_tablet_manager.cc:1397] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260929 02:46:06.622668 2649 tablet_bootstrap.cc:492] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: No bootstrap required, opened a new log
I20260929 02:46:06.622800 2649 ts_tablet_manager.cc:1397] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260929 02:46:06.623034 2650 raft_consensus.cc:348] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.623263 2650 raft_consensus.cc:374] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.623361 2650 raft_consensus.cc:729] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.623553 2649 raft_consensus.cc:348] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.623589 2650 consensus_queue.cc:260] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.623796 2649 raft_consensus.cc:374] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.623885 2649 raft_consensus.cc:729] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f003bde72b74706a5fffad865915b35, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.624168 2650 ts_tablet_manager.cc:1428] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:46:06.624100 2649 consensus_queue.cc:260] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.624465 2656 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.624770 2656 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:06.624504 2655 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.624859 2655 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:06.625277 2649 ts_tablet_manager.cc:1428] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:46:06.731290 2656 raft_consensus.cc:482] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:06.731490 2656 raft_consensus.cc:504] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.732126 2656 leader_election.cc:288] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:06.732659 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a56464fc104fe3b986f6a865a04500" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37" is_pre_election: true
I20260929 02:46:06.732684 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a56464fc104fe3b986f6a865a04500" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f003bde72b74706a5fffad865915b35" is_pre_election: true
I20260929 02:46:06.732903 2580 raft_consensus.cc:2454] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 0.
I20260929 02:46:06.732954 2445 raft_consensus.cc:2454] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 0.
I20260929 02:46:06.733322 2233 leader_election.cc:302] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 763f17acab1941d792bfa92beeb2162d, a4edb27278484b4cb4e7945424cfaf37; no voters:
I20260929 02:46:06.733601 2656 raft_consensus.cc:2781] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:06.733714 2656 raft_consensus.cc:482] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:06.733820 2656 raft_consensus.cc:3037] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:06.735807 2656 raft_consensus.cc:504] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.736382 2656 leader_election.cc:288] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 election: Requested vote from peers 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:06.736853 2445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a56464fc104fe3b986f6a865a04500" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f003bde72b74706a5fffad865915b35"
I20260929 02:46:06.736999 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90a56464fc104fe3b986f6a865a04500" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37"
I20260929 02:46:06.737097 2445 raft_consensus.cc:3037] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:06.737232 2580 raft_consensus.cc:3037] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:06.739198 2445 raft_consensus.cc:2454] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 1.
I20260929 02:46:06.739573 2235 leader_election.cc:302] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f003bde72b74706a5fffad865915b35, 763f17acab1941d792bfa92beeb2162d; no voters:
I20260929 02:46:06.739660 2580 raft_consensus.cc:2454] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 1.
I20260929 02:46:06.739887 2656 raft_consensus.cc:2781] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:06.740114 2656 raft_consensus.cc:686] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [term 1 LEADER]: Becoming Leader. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Running, Role: LEADER
I20260929 02:46:06.740420 2656 consensus_queue.cc:237] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.742223 2660 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: TxnStatusTablet state changed. Reason: New leader 763f17acab1941d792bfa92beeb2162d. Latest consensus state: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.742681 2660 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:06.743321 2672 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:46:06.743597 2672 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:46:06.743600 2144 catalog_manager.cc:5495] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d reported cstate change: term changed from 0 to 1, leader changed from <none> to 763f17acab1941d792bfa92beeb2162d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:06.789783 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.782238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.0.97.1:42441)
0929 02:46:06.782456 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.782463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.782681 (+ 218us) server_negotiation.cc:407] Connection header received
0929 02:46:06.782944 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.782954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.783076 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.783233 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:06.783597 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.784755 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.786659 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.787287 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.787483 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.788204 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:06.789017 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:06.789204 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.789364 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.789445 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:46:06.789903 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.782218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42441 (local address 127.0.0.1:47944)
0929 02:46:06.782580 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.782599 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.782740 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.783257 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.783261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.783280 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:06.783463 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.783472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.784899 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.784905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.786505 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.786519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.787683 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.787690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.787928 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.787997 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:06.789160 (+ 1163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:06.789190 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.789333 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.789501 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260929 02:46:06.798643 2580 raft_consensus.cc:1261] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Refusing update from remote peer 763f17acab1941d792bfa92beeb2162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:06.798755 2445 raft_consensus.cc:1261] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Refusing update from remote peer 763f17acab1941d792bfa92beeb2162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:06.799624 2660 consensus_queue.cc:1035] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:06.799994 2656 consensus_queue.cc:1035] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:06.811167 2655 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: TxnStatusTablet state changed. Reason: New leader 763f17acab1941d792bfa92beeb2162d. Latest consensus state: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.811455 2655 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:06.818476 2675 mvcc.cc:205] Tried to move back new op lower bound from 7334502263997677568 to 7334502263771500544. Current Snapshot: MvccSnapshot[applied={T|T < 7334502263997677568}]
I20260929 02:46:06.818831 2656 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.819211 2656 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:06.829679 2655 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.829917 2655 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P 1f003bde72b74706a5fffad865915b35: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:06.829818 2657 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: TxnStatusTablet state changed. Reason: New leader 763f17acab1941d792bfa92beeb2162d. Latest consensus state: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.830132 2657 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:06.828457 2656 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.832186 2656 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P 763f17acab1941d792bfa92beeb2162d: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:06.835309 2657 tablet_replica.cc:439] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } } }
I20260929 02:46:06.835528 2657 tablet_replica.cc:442] T 90a56464fc104fe3b986f6a865a04500 P a4edb27278484b4cb4e7945424cfaf37: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:06.835698 2677 mvcc.cc:205] Tried to move back new op lower bound from 7334502263997677568 to 7334502263771500544. Current Snapshot: MvccSnapshot[applied={T|T < 7334502263997677568 or (T in {7334502263997677568})}]
I20260929 02:46:06.843942 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.837677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.0.97.1:42441)
0929 02:46:06.837854 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.837862 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.838177 (+ 315us) server_negotiation.cc:407] Connection header received
0929 02:46:06.838342 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.838350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.838422 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.838511 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:06.838951 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.839897 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.841934 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.842415 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.842527 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.843063 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:06.843216 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:06.843309 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.843477 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.843550 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:46:06.862282 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.855444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48499 (local address 127.0.97.1:42441)
0929 02:46:06.855647 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:06.855653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:06.856278 (+ 625us) server_negotiation.cc:407] Connection header received
0929 02:46:06.856418 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:06.856423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:06.856508 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:06.856620 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:06.857145 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.858281 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.859785 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.860505 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.860657 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.861366 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:06.861492 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:06.861609 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:06.861824 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:46:06.861955 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":54}
I20260929 02:46:06.862561 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:06.855428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42441 (local address 127.0.97.1:48499)
0929 02:46:06.856163 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:06.856184 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:06.856306 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:06.856651 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:06.856655 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:06.856698 (+ 43us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:06.856993 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.857002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.858426 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.858430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.859619 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:06.859630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:06.860833 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:06.860839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:06.861023 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:06.861110 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:06.861648 (+ 538us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:06.861662 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 02:46:06.861772 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:46:06.861909 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":3,"thread_start_us":394,"threads_started":1}
I20260929 02:46:06.872335 2579 raft_consensus.cc:1261] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Refusing update from remote peer 763f17acab1941d792bfa92beeb2162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:06.872918 2445 raft_consensus.cc:1261] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Refusing update from remote peer 763f17acab1941d792bfa92beeb2162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:06.872907 2660 consensus_queue.cc:1035] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:06.873431 2660 consensus_queue.cc:1035] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:06.940177 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:06.940416 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:06.943945 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Sending DeleteTablet for 3 replicas of tablet b519126285964517985b244b217ac1e0
I20260929 02:46:06.944957 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet b519126285964517985b244b217ac1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47934
I20260929 02:46:06.945067 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet b519126285964517985b244b217ac1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59136
I20260929 02:46:06.945855 2691 tablet_replica.cc:330] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: stopping tablet replica
I20260929 02:46:06.946293 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet b519126285964517985b244b217ac1e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:06 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37170
I20260929 02:46:06.946784 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 02:46:06.947189 2693 tablet_replica.cc:330] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: stopping tablet replica
W20260929 02:46:06.947525 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:06.947570 2693 raft_consensus.cc:2229] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:06.947898 2693 raft_consensus.cc:2258] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:06.946377 2691 raft_consensus.cc:2229] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:06.949787 2693 ts_tablet_manager.cc:1905] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:06.949841 2691 raft_consensus.cc:2258] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:06.951782 2691 ts_tablet_manager.cc:1905] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:06.956348 2282 tablet_service.cc:1467] Processing CreateTablet for tablet ede3b47a0a5e44888dda7334f774bb7b (DEFAULT_TABLE table=test-table [id=ce2173a3806745a1988cbf7e0fc03225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:06.956817 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede3b47a0a5e44888dda7334f774bb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.956852 2560 tablet_service.cc:1467] Processing CreateTablet for tablet ede3b47a0a5e44888dda7334f774bb7b (DEFAULT_TABLE table=test-table [id=ce2173a3806745a1988cbf7e0fc03225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:06.957230 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede3b47a0a5e44888dda7334f774bb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.960379 2421 tablet_service.cc:1467] Processing CreateTablet for tablet ede3b47a0a5e44888dda7334f774bb7b (DEFAULT_TABLE table=test-table [id=ce2173a3806745a1988cbf7e0fc03225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:06.961795 2692 tablet_replica.cc:330] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: stopping tablet replica
I20260929 02:46:06.962033 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ede3b47a0a5e44888dda7334f774bb7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:06.962432 2692 raft_consensus.cc:2229] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:06.962842 2692 raft_consensus.cc:2258] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:06.963734 2650 tablet_bootstrap.cc:492] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: Bootstrap starting.
I20260929 02:46:06.965329 2651 tablet_bootstrap.cc:492] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: Bootstrap starting.
I20260929 02:46:06.965418 2650 tablet_bootstrap.cc:654] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.966902 2651 tablet_bootstrap.cc:654] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.965317 2693 ts_tablet_manager.cc:1918] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:06.971374 2693 log.cc:1196] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/b519126285964517985b244b217ac1e0
I20260929 02:46:06.971899 2693 ts_tablet_manager.cc:1939] T b519126285964517985b244b217ac1e0 P a4edb27278484b4cb4e7945424cfaf37: Deleting consensus metadata
I20260929 02:46:06.973838 2130 catalog_manager.cc:4841] TS a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519): tablet b519126285964517985b244b217ac1e0 (table test-table [id=bc9f5959e27741b783d382134d9615c9]) successfully deleted
I20260929 02:46:06.974202 2649 tablet_bootstrap.cc:492] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: Bootstrap starting.
I20260929 02:46:06.975941 2692 ts_tablet_manager.cc:1905] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:06.976490 2649 tablet_bootstrap.cc:654] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:06.980448 2650 tablet_bootstrap.cc:492] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: No bootstrap required, opened a new log
I20260929 02:46:06.980741 2650 ts_tablet_manager.cc:1397] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260929 02:46:06.980938 2651 tablet_bootstrap.cc:492] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: No bootstrap required, opened a new log
I20260929 02:46:06.981106 2651 ts_tablet_manager.cc:1397] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.006s
I20260929 02:46:06.981737 2650 raft_consensus.cc:348] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.981827 2651 raft_consensus.cc:348] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.982076 2651 raft_consensus.cc:374] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.982107 2650 raft_consensus.cc:374] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.982156 2651 raft_consensus.cc:729] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4edb27278484b4cb4e7945424cfaf37, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.982297 2650 raft_consensus.cc:729] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.982422 2651 consensus_queue.cc:260] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.982748 2650 consensus_queue.cc:260] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.982952 2651 ts_tablet_manager.cc:1428] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:06.983716 2691 ts_tablet_manager.cc:1918] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:06.983915 2691 log.cc:1196] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/b519126285964517985b244b217ac1e0
I20260929 02:46:06.984679 2649 tablet_bootstrap.cc:492] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: No bootstrap required, opened a new log
I20260929 02:46:06.984865 2649 ts_tablet_manager.cc:1397] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260929 02:46:06.985206 2691 ts_tablet_manager.cc:1939] T b519126285964517985b244b217ac1e0 P 763f17acab1941d792bfa92beeb2162d: Deleting consensus metadata
I20260929 02:46:06.985627 2649 raft_consensus.cc:348] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.985862 2649 raft_consensus.cc:374] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:06.985936 2649 raft_consensus.cc:729] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f003bde72b74706a5fffad865915b35, State: Initialized, Role: FOLLOWER
I20260929 02:46:06.986260 2649 consensus_queue.cc:260] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:06.986461 2692 ts_tablet_manager.cc:1918] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:06.986600 2692 log.cc:1196] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/b519126285964517985b244b217ac1e0
I20260929 02:46:06.987140 2692 ts_tablet_manager.cc:1939] T b519126285964517985b244b217ac1e0 P 1f003bde72b74706a5fffad865915b35: Deleting consensus metadata
I20260929 02:46:06.987608 2132 catalog_manager.cc:4841] TS 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441): tablet b519126285964517985b244b217ac1e0 (table test-table [id=bc9f5959e27741b783d382134d9615c9]) successfully deleted
I20260929 02:46:06.988117 2649 ts_tablet_manager.cc:1428] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:06.987596 2650 ts_tablet_manager.cc:1428] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260929 02:46:06.988801 2132 catalog_manager.cc:4841] TS 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825): tablet b519126285964517985b244b217ac1e0 (table test-table [id=bc9f5959e27741b783d382134d9615c9]) successfully deleted
I20260929 02:46:07.050717 2655 raft_consensus.cc:482] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:07.050909 2655 raft_consensus.cc:504] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:07.052368 2655 leader_election.cc:288] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:07.060669 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.053231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52771 (local address 127.0.97.3:41519)
0929 02:46:07.054211 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:07.054218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:07.054235 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:46:07.054321 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:07.054326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:07.054413 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:07.054523 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:07.054974 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.055964 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.058327 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.059535 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.059670 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.059778 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:07.060154 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 02:46:07.060320 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":874,"thread_start_us":755,"threads_started":1}
I20260929 02:46:07.060739 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.052863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55957 (local address 127.0.97.1:42441)
0929 02:46:07.053056 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:07.053063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:07.054126 (+ 1063us) server_negotiation.cc:407] Connection header received
0929 02:46:07.054390 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:07.054399 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:07.054490 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:07.054593 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:07.055425 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.056370 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.058156 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.059535 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.059679 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.059785 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:07.060318 (+ 533us) server_negotiation.cc:299] Negotiation successful
0929 02:46:07.060448 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:46:07.061147 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.052588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41519 (local address 127.0.97.2:52771)
0929 02:46:07.053196 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:07.053298 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:07.053489 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:07.054548 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:07.054553 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:07.054577 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:07.054815 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.054824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.056102 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.056108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.058154 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.058167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.059715 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.059721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.059942 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.060009 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 02:46:07.060111 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:46:07.060263 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":8,"thread_start_us":425,"threads_started":1}
I20260929 02:46:07.061710 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.052594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42441 (local address 127.0.97.2:55957)
0929 02:46:07.054025 (+ 1431us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:07.054044 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:07.054211 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:07.054748 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:07.054755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:07.054790 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:07.055244 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.055254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.056501 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.056508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.057995 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.058006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.059939 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.059943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.060140 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.060173 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:46:07.060959 (+ 786us) client_negotiation.cc:240] Negotiation successful
0929 02:46:07.061086 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1308,"mutex_wait_us":272,"thread_start_us":901,"threads_started":1}
I20260929 02:46:07.061909 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede3b47a0a5e44888dda7334f774bb7b" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37" is_pre_election: true
I20260929 02:46:07.062224 2580 raft_consensus.cc:2454] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 0.
I20260929 02:46:07.062620 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede3b47a0a5e44888dda7334f774bb7b" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763f17acab1941d792bfa92beeb2162d" is_pre_election: true
I20260929 02:46:07.062968 2373 leader_election.cc:302] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f003bde72b74706a5fffad865915b35, a4edb27278484b4cb4e7945424cfaf37; no voters:
I20260929 02:46:07.063050 2302 raft_consensus.cc:2454] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 0.
I20260929 02:46:07.063380 2655 raft_consensus.cc:2781] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:07.063491 2655 raft_consensus.cc:482] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:07.063583 2655 raft_consensus.cc:3037] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:07.066622 2655 raft_consensus.cc:504] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:07.067394 2655 leader_election.cc:288] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 election: Requested vote from peers 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:07.067857 2302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede3b47a0a5e44888dda7334f774bb7b" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763f17acab1941d792bfa92beeb2162d"
I20260929 02:46:07.067868 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ede3b47a0a5e44888dda7334f774bb7b" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37"
I20260929 02:46:07.068058 2302 raft_consensus.cc:3037] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:07.068130 2580 raft_consensus.cc:3037] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:07.070343 2302 raft_consensus.cc:2454] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 1.
I20260929 02:46:07.070788 2375 leader_election.cc:302] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f003bde72b74706a5fffad865915b35, 763f17acab1941d792bfa92beeb2162d; no voters:
I20260929 02:46:07.071058 2655 raft_consensus.cc:2781] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:07.071058 2580 raft_consensus.cc:2454] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 1.
I20260929 02:46:07.071679 2655 raft_consensus.cc:686] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 1 LEADER]: Becoming Leader. State: Replica: 1f003bde72b74706a5fffad865915b35, State: Running, Role: LEADER
I20260929 02:46:07.072068 2655 consensus_queue.cc:237] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:07.075510 2143 catalog_manager.cc:5495] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f003bde72b74706a5fffad865915b35 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1f003bde72b74706a5fffad865915b35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:07.107028 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.100770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59138 (local address 127.0.97.2:39825)
0929 02:46:07.101803 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:07.101811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:07.101837 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:46:07.101914 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:07.101918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:07.101987 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:07.102156 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:07.102592 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.103446 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.104784 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.105183 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.105335 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.105794 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:07.106375 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:07.106531 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:07.106613 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 02:46:07.106697 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":755,"threads_started":1}
I20260929 02:46:07.124068 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.118003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34125 (local address 127.0.97.1:42441)
0929 02:46:07.118216 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:07.118224 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:07.118856 (+ 632us) server_negotiation.cc:407] Connection header received
0929 02:46:07.119138 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:07.119144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:07.119224 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:07.119321 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:07.119848 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.120751 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.122239 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.122709 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.122852 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.123280 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:07.123407 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:07.123505 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:07.123647 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 02:46:07.123735 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 02:46:07.124128 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.117996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42441 (local address 127.0.97.2:34125)
0929 02:46:07.118657 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:07.118677 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:07.118980 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:07.119345 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:07.119349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:07.119360 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:07.119689 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.119697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.120877 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.120885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.122095 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.122107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.122908 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.122915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.123043 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.123097 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:07.123505 (+ 408us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:07.123519 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 02:46:07.123621 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 02:46:07.123756 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":382,"threads_started":1}
I20260929 02:46:07.133800 2580 raft_consensus.cc:1261] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Refusing update from remote peer 1f003bde72b74706a5fffad865915b35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:07.134143 2302 raft_consensus.cc:1261] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Refusing update from remote peer 1f003bde72b74706a5fffad865915b35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:07.134492 2700 consensus_queue.cc:1035] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:07.134869 2655 consensus_queue.cc:1035] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:07.142468 2676 mvcc.cc:205] Tried to move back new op lower bound from 7334502265374138368 to 7334502265130860544. Current Snapshot: MvccSnapshot[applied={T|T < 7334502265374138368}]
I20260929 02:46:07.167189 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.161242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54899 (local address 127.0.97.2:39825)
0929 02:46:07.161410 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:07.161418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:07.161465 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:46:07.161672 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:07.161678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:07.161765 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:07.161839 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:07.162250 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.163093 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.165257 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.165907 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.166040 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.166494 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:07.166618 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:07.166717 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:07.166892 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 02:46:07.166957 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:46:07.167358 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:07.161183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39825 (local address 127.0.97.1:54899)
0929 02:46:07.161382 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:07.161404 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:07.161532 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:07.161880 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:07.161885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:07.161898 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:07.162103 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.162114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.163258 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.163264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.165084 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:07.165103 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:07.166073 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:07.166078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:07.166286 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:07.166337 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:07.166749 (+ 412us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:07.166762 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 02:46:07.166855 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:46:07.167011 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 02:46:07.168380 2676 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:46:08.771273 2144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:08.771548 2144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:08.775609 2144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Sending DeleteTablet for 3 replicas of tablet ede3b47a0a5e44888dda7334f774bb7b
I20260929 02:46:08.776664 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet ede3b47a0a5e44888dda7334f774bb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37170
I20260929 02:46:08.776820 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet ede3b47a0a5e44888dda7334f774bb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59136
I20260929 02:46:08.780797 2282 tablet_service.cc:1514] Processing DeleteTablet for tablet ede3b47a0a5e44888dda7334f774bb7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47934
I20260929 02:46:08.789844 2732 tablet_replica.cc:330] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: stopping tablet replica
I20260929 02:46:08.790078 2732 raft_consensus.cc:2229] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:08.790656 2734 tablet_replica.cc:330] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: stopping tablet replica
I20260929 02:46:08.790875 2734 raft_consensus.cc:2229] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:08.791003 2732 raft_consensus.cc:2258] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:08.791098 2734 raft_consensus.cc:2258] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:08.792265 2733 tablet_replica.cc:330] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: stopping tablet replica
I20260929 02:46:08.792253 2734 ts_tablet_manager.cc:1905] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:08.792285 2732 ts_tablet_manager.cc:1905] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:08.792470 2733 raft_consensus.cc:2229] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:08.793033 2733 raft_consensus.cc:2258] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:08.793895 2733 ts_tablet_manager.cc:1905] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:08.799060 2732 ts_tablet_manager.cc:1918] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:46:08.799203 2732 log.cc:1196] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/ede3b47a0a5e44888dda7334f774bb7b
I20260929 02:46:08.799599 2732 ts_tablet_manager.cc:1939] T ede3b47a0a5e44888dda7334f774bb7b P a4edb27278484b4cb4e7945424cfaf37: Deleting consensus metadata
I20260929 02:46:08.800712 2130 catalog_manager.cc:4841] TS a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519): tablet ede3b47a0a5e44888dda7334f774bb7b (table test-table [id=ce2173a3806745a1988cbf7e0fc03225]) successfully deleted
I20260929 02:46:08.801411 2733 ts_tablet_manager.cc:1918] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:46:08.801551 2733 log.cc:1196] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/ede3b47a0a5e44888dda7334f774bb7b
I20260929 02:46:08.802042 2733 ts_tablet_manager.cc:1939] T ede3b47a0a5e44888dda7334f774bb7b P 1f003bde72b74706a5fffad865915b35: Deleting consensus metadata
I20260929 02:46:08.803249 2132 catalog_manager.cc:4841] TS 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825): tablet ede3b47a0a5e44888dda7334f774bb7b (table test-table [id=ce2173a3806745a1988cbf7e0fc03225]) successfully deleted
I20260929 02:46:08.804145 2734 ts_tablet_manager.cc:1918] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:46:08.804272 2734 log.cc:1196] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/ede3b47a0a5e44888dda7334f774bb7b
I20260929 02:46:08.804720 2734 ts_tablet_manager.cc:1939] T ede3b47a0a5e44888dda7334f774bb7b P 763f17acab1941d792bfa92beeb2162d: Deleting consensus metadata
I20260929 02:46:08.805743 2132 catalog_manager.cc:4841] TS 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441): tablet ede3b47a0a5e44888dda7334f774bb7b (table test-table [id=ce2173a3806745a1988cbf7e0fc03225]) successfully deleted
Sep 29 02:46:08 dist-test-slave-6tjh krb5kdc[2100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:46:08 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:08.841324 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:08.827280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.0.97.62:36921)
0929 02:46:08.827923 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:08.827929 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:08.827956 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:46:08.828137 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:08.828141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:08.828205 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:08.828318 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:08.829753 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:08.830525 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:08.831790 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:08.832171 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:08.832308 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:08.836058 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:08.836063 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:08.836066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:08.836097 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:08.838767 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:08.839463 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:08.839467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:08.839469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:08.839591 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:08.839928 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:08.839931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:08.839933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:08.840277 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:08.840517 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:08.840737 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:46:08.840899 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":320,"threads_started":1}
I20260929 02:46:08.844492 2144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:08.845108 2144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:08.851362 2282 tablet_service.cc:1467] Processing CreateTablet for tablet f420105c2e1e4e9da637762966361b1a (DEFAULT_TABLE table=test-table [id=699506217a27407d94389b316f1a5f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:08.851449 2421 tablet_service.cc:1467] Processing CreateTablet for tablet f420105c2e1e4e9da637762966361b1a (DEFAULT_TABLE table=test-table [id=699506217a27407d94389b316f1a5f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:08.851804 2282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f420105c2e1e4e9da637762966361b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:08.851711 2560 tablet_service.cc:1467] Processing CreateTablet for tablet f420105c2e1e4e9da637762966361b1a (DEFAULT_TABLE table=test-table [id=699506217a27407d94389b316f1a5f80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:08.852317 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f420105c2e1e4e9da637762966361b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:08.852516 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f420105c2e1e4e9da637762966361b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:08.860365 2745 tablet_bootstrap.cc:492] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: Bootstrap starting.
I20260929 02:46:08.861263 2743 tablet_bootstrap.cc:492] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: Bootstrap starting.
I20260929 02:46:08.861719 2744 tablet_bootstrap.cc:492] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: Bootstrap starting.
I20260929 02:46:08.862860 2745 tablet_bootstrap.cc:654] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:08.863435 2743 tablet_bootstrap.cc:654] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:08.863641 2744 tablet_bootstrap.cc:654] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:08.865546 2743 tablet_bootstrap.cc:492] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: No bootstrap required, opened a new log
I20260929 02:46:08.865545 2745 tablet_bootstrap.cc:492] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: No bootstrap required, opened a new log
I20260929 02:46:08.865711 2743 ts_tablet_manager.cc:1397] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:46:08.865715 2745 ts_tablet_manager.cc:1397] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:08.865923 2744 tablet_bootstrap.cc:492] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: No bootstrap required, opened a new log
I20260929 02:46:08.866056 2744 ts_tablet_manager.cc:1397] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:46:08.866530 2745 raft_consensus.cc:348] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.866533 2743 raft_consensus.cc:348] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.866767 2743 raft_consensus.cc:374] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:08.866780 2745 raft_consensus.cc:374] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:08.866878 2745 raft_consensus.cc:729] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4edb27278484b4cb4e7945424cfaf37, State: Initialized, Role: FOLLOWER
I20260929 02:46:08.866776 2744 raft_consensus.cc:348] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.867000 2744 raft_consensus.cc:374] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:08.867082 2744 raft_consensus.cc:729] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f003bde72b74706a5fffad865915b35, State: Initialized, Role: FOLLOWER
I20260929 02:46:08.867108 2745 consensus_queue.cc:260] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.867307 2743 raft_consensus.cc:729] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Initialized, Role: FOLLOWER
I20260929 02:46:08.867339 2744 consensus_queue.cc:260] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.867990 2744 ts_tablet_manager.cc:1428] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:08.868115 2745 ts_tablet_manager.cc:1428] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:08.867542 2743 consensus_queue.cc:260] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.868887 2743 ts_tablet_manager.cc:1428] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 02:46:08.956576 2710 raft_consensus.cc:482] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:08.956820 2710 raft_consensus.cc:504] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.957603 2710 leader_election.cc:288] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:08.958171 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f420105c2e1e4e9da637762966361b1a" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37" is_pre_election: true
I20260929 02:46:08.958145 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f420105c2e1e4e9da637762966361b1a" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f003bde72b74706a5fffad865915b35" is_pre_election: true
I20260929 02:46:08.958427 2444 raft_consensus.cc:2454] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 0.
I20260929 02:46:08.958439 2580 raft_consensus.cc:2454] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 0.
I20260929 02:46:08.958923 2233 leader_election.cc:302] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 763f17acab1941d792bfa92beeb2162d, a4edb27278484b4cb4e7945424cfaf37; no voters:
I20260929 02:46:08.959192 2710 raft_consensus.cc:2781] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:08.959295 2710 raft_consensus.cc:482] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:08.959409 2710 raft_consensus.cc:3037] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:08.961668 2710 raft_consensus.cc:504] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.962270 2710 leader_election.cc:288] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 election: Requested vote from peers 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:08.962740 2580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f420105c2e1e4e9da637762966361b1a" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37"
I20260929 02:46:08.962759 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f420105c2e1e4e9da637762966361b1a" candidate_uuid: "763f17acab1941d792bfa92beeb2162d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f003bde72b74706a5fffad865915b35"
I20260929 02:46:08.962973 2444 raft_consensus.cc:3037] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:08.962991 2580 raft_consensus.cc:3037] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:08.965916 2580 raft_consensus.cc:2454] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 1.
I20260929 02:46:08.965931 2444 raft_consensus.cc:2454] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 763f17acab1941d792bfa92beeb2162d in term 1.
I20260929 02:46:08.966384 2235 leader_election.cc:302] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f003bde72b74706a5fffad865915b35, 763f17acab1941d792bfa92beeb2162d; no voters:
I20260929 02:46:08.966683 2710 raft_consensus.cc:2781] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:08.966846 2710 raft_consensus.cc:686] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 1 LEADER]: Becoming Leader. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Running, Role: LEADER
I20260929 02:46:08.967125 2710 consensus_queue.cc:237] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:08.969879 2145 catalog_manager.cc:5495] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d reported cstate change: term changed from 0 to 1, leader changed from <none> to 763f17acab1941d792bfa92beeb2162d (127.0.97.1). New cstate: current_term: 1 leader_uuid: "763f17acab1941d792bfa92beeb2162d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:09.001415 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:08.996771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.0.97.1:42441)
0929 02:46:08.997303 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:08.997307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:08.997325 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:46:08.997371 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:08.997375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:08.997430 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:08.997509 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:08.997850 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:08.998588 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:08.999831 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:09.000242 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:09.000333 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:09.000696 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:09.000827 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:09.000896 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:09.001009 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 02:46:09.001063 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":323,"threads_started":1}
I20260929 02:46:09.010012 2580 raft_consensus.cc:1261] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Refusing update from remote peer 763f17acab1941d792bfa92beeb2162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:09.010032 2444 raft_consensus.cc:1261] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Refusing update from remote peer 763f17acab1941d792bfa92beeb2162d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:09.010551 2718 consensus_queue.cc:1035] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:09.010845 2710 consensus_queue.cc:1035] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:09.052937 2143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:09.053157 2143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:09.056478 2143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Sending DeleteTablet for 3 replicas of tablet f420105c2e1e4e9da637762966361b1a
I20260929 02:46:09.057556 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet f420105c2e1e4e9da637762966361b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47934
I20260929 02:46:09.057901 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet f420105c2e1e4e9da637762966361b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37170
I20260929 02:46:09.057947 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet f420105c2e1e4e9da637762966361b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:09 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59136
I20260929 02:46:09.058228 2734 tablet_replica.cc:330] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: stopping tablet replica
I20260929 02:46:09.058327 2732 tablet_replica.cc:330] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: stopping tablet replica
I20260929 02:46:09.058442 2734 raft_consensus.cc:2229] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:09.058579 2732 raft_consensus.cc:2229] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:09.058727 2734 raft_consensus.cc:2258] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:09.058815 2732 raft_consensus.cc:2258] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:09.058818 2143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 02:46:09.059114 2733 tablet_replica.cc:330] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: stopping tablet replica
I20260929 02:46:09.059316 2733 raft_consensus.cc:2229] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 02:46:09.059500 2143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:09.059530 2733 raft_consensus.cc:2258] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:09.059582 2734 ts_tablet_manager.cc:1905] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:09.060760 2732 ts_tablet_manager.cc:1905] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:09.065590 2734 ts_tablet_manager.cc:1918] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:09.065747 2734 log.cc:1196] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/f420105c2e1e4e9da637762966361b1a
I20260929 02:46:09.066191 2734 ts_tablet_manager.cc:1939] T f420105c2e1e4e9da637762966361b1a P 763f17acab1941d792bfa92beeb2162d: Deleting consensus metadata
I20260929 02:46:09.067248 2281 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2df21eb0ab467690404bfcec9b4a3d (DEFAULT_TABLE table=test-table [id=4eb19502dbac452383a72c3515c089e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:09.067662 2281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2df21eb0ab467690404bfcec9b4a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:09.067260 2560 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2df21eb0ab467690404bfcec9b4a3d (DEFAULT_TABLE table=test-table [id=4eb19502dbac452383a72c3515c089e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:09.068336 2560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2df21eb0ab467690404bfcec9b4a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:09.068393 2733 ts_tablet_manager.cc:1905] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:09.072901 2421 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2df21eb0ab467690404bfcec9b4a3d (DEFAULT_TABLE table=test-table [id=4eb19502dbac452383a72c3515c089e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:09.073316 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c2df21eb0ab467690404bfcec9b4a3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:09.066977 2732 ts_tablet_manager.cc:1918] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:09.075260 2732 log.cc:1196] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/f420105c2e1e4e9da637762966361b1a
I20260929 02:46:09.075716 2732 ts_tablet_manager.cc:1939] T f420105c2e1e4e9da637762966361b1a P a4edb27278484b4cb4e7945424cfaf37: Deleting consensus metadata
I20260929 02:46:09.076028 2733 ts_tablet_manager.cc:1918] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:09.076182 2733 log.cc:1196] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/f420105c2e1e4e9da637762966361b1a
I20260929 02:46:09.076604 2733 ts_tablet_manager.cc:1939] T f420105c2e1e4e9da637762966361b1a P 1f003bde72b74706a5fffad865915b35: Deleting consensus metadata
I20260929 02:46:09.076794 2745 tablet_bootstrap.cc:492] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: Bootstrap starting.
I20260929 02:46:09.076792 2130 catalog_manager.cc:4841] TS a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519): tablet f420105c2e1e4e9da637762966361b1a (table test-table [id=699506217a27407d94389b316f1a5f80]) successfully deleted
I20260929 02:46:09.077090 2743 tablet_bootstrap.cc:492] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: Bootstrap starting.
I20260929 02:46:09.077777 2132 catalog_manager.cc:4841] TS 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825): tablet f420105c2e1e4e9da637762966361b1a (table test-table [id=699506217a27407d94389b316f1a5f80]) successfully deleted
I20260929 02:46:09.078436 2745 tablet_bootstrap.cc:654] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:09.078817 2743 tablet_bootstrap.cc:654] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:09.080907 2743 tablet_bootstrap.cc:492] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: No bootstrap required, opened a new log
I20260929 02:46:09.081369 2743 ts_tablet_manager.cc:1397] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260929 02:46:09.081599 2132 catalog_manager.cc:4841] TS 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441): tablet f420105c2e1e4e9da637762966361b1a (table test-table [id=699506217a27407d94389b316f1a5f80]) successfully deleted
I20260929 02:46:09.082229 2743 raft_consensus.cc:348] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.082444 2744 tablet_bootstrap.cc:492] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: Bootstrap starting.
I20260929 02:46:09.083140 2743 raft_consensus.cc:374] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:09.083312 2743 raft_consensus.cc:729] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 763f17acab1941d792bfa92beeb2162d, State: Initialized, Role: FOLLOWER
I20260929 02:46:09.083921 2745 tablet_bootstrap.cc:492] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: No bootstrap required, opened a new log
I20260929 02:46:09.084084 2745 ts_tablet_manager.cc:1397] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260929 02:46:09.083774 2743 consensus_queue.cc:260] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.084628 2744 tablet_bootstrap.cc:654] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:09.084692 2743 ts_tablet_manager.cc:1428] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:09.084888 2745 raft_consensus.cc:348] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.085157 2745 raft_consensus.cc:374] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:09.085248 2745 raft_consensus.cc:729] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4edb27278484b4cb4e7945424cfaf37, State: Initialized, Role: FOLLOWER
I20260929 02:46:09.085541 2745 consensus_queue.cc:260] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.086292 2745 ts_tablet_manager.cc:1428] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:09.087240 2744 tablet_bootstrap.cc:492] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: No bootstrap required, opened a new log
I20260929 02:46:09.087423 2744 ts_tablet_manager.cc:1397] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:46:09.088300 2744 raft_consensus.cc:348] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.088603 2744 raft_consensus.cc:374] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:09.088696 2744 raft_consensus.cc:729] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f003bde72b74706a5fffad865915b35, State: Initialized, Role: FOLLOWER
I20260929 02:46:09.088960 2744 consensus_queue.cc:260] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.089584 2744 ts_tablet_manager.cc:1428] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:09.208741 2723 raft_consensus.cc:482] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:09.208969 2723 raft_consensus.cc:504] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.209723 2723 leader_election.cc:288] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:09.210318 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df21eb0ab467690404bfcec9b4a3d" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37" is_pre_election: true
I20260929 02:46:09.210338 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df21eb0ab467690404bfcec9b4a3d" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763f17acab1941d792bfa92beeb2162d" is_pre_election: true
I20260929 02:46:09.210604 2301 raft_consensus.cc:2454] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 0.
I20260929 02:46:09.210635 2579 raft_consensus.cc:2454] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 0.
I20260929 02:46:09.211099 2373 leader_election.cc:302] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f003bde72b74706a5fffad865915b35, a4edb27278484b4cb4e7945424cfaf37; no voters:
I20260929 02:46:09.211405 2723 raft_consensus.cc:2781] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:09.211513 2723 raft_consensus.cc:482] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:09.211619 2723 raft_consensus.cc:3037] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:09.213692 2723 raft_consensus.cc:504] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.214226 2723 leader_election.cc:288] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 election: Requested vote from peers 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441), a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519)
I20260929 02:46:09.214787 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df21eb0ab467690404bfcec9b4a3d" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4edb27278484b4cb4e7945424cfaf37"
I20260929 02:46:09.214839 2301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c2df21eb0ab467690404bfcec9b4a3d" candidate_uuid: "1f003bde72b74706a5fffad865915b35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "763f17acab1941d792bfa92beeb2162d"
I20260929 02:46:09.215026 2579 raft_consensus.cc:3037] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:09.215047 2301 raft_consensus.cc:3037] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:09.218394 2579 raft_consensus.cc:2454] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 1.
I20260929 02:46:09.218410 2301 raft_consensus.cc:2454] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f003bde72b74706a5fffad865915b35 in term 1.
I20260929 02:46:09.218901 2373 leader_election.cc:302] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f003bde72b74706a5fffad865915b35, 763f17acab1941d792bfa92beeb2162d, a4edb27278484b4cb4e7945424cfaf37; no voters:
I20260929 02:46:09.219252 2723 raft_consensus.cc:2781] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:09.219421 2723 raft_consensus.cc:686] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 1 LEADER]: Becoming Leader. State: Replica: 1f003bde72b74706a5fffad865915b35, State: Running, Role: LEADER
I20260929 02:46:09.219751 2723 consensus_queue.cc:237] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } }
I20260929 02:46:09.222388 2145 catalog_manager.cc:5495] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f003bde72b74706a5fffad865915b35 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "1f003bde72b74706a5fffad865915b35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f003bde72b74706a5fffad865915b35" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39825 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:09.242985 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:09.237716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.0.97.2:39825)
0929 02:46:09.238297 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:09.238301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:09.238320 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:46:09.238366 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:09.238370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:09.238425 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:09.238516 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:09.238844 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:09.239601 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:09.241047 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:09.241686 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:09.241784 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:09.242230 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:09.242337 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:09.242423 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:09.242537 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 02:46:09.242652 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":335,"threads_started":1}
I20260929 02:46:09.251194 2579 raft_consensus.cc:1261] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Refusing update from remote peer 1f003bde72b74706a5fffad865915b35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:09.251293 2301 raft_consensus.cc:1261] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Refusing update from remote peer 1f003bde72b74706a5fffad865915b35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:09.251713 2726 consensus_queue.cc:1035] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4edb27278484b4cb4e7945424cfaf37" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:09.252022 2723 consensus_queue.cc:1035] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "763f17acab1941d792bfa92beeb2162d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:46:09.281991 2715 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 02:46:10.859613 2145 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:10.859902 2145 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:10.863821 2145 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c325640a5e594d6280f5b29f7aee512e: Sending DeleteTablet for 3 replicas of tablet 5c2df21eb0ab467690404bfcec9b4a3d
I20260929 02:46:10.864562 2281 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2df21eb0ab467690404bfcec9b4a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47934
I20260929 02:46:10.864679 2421 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2df21eb0ab467690404bfcec9b4a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59136
I20260929 02:46:10.864727 2560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2df21eb0ab467690404bfcec9b4a3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37170
I20260929 02:46:10.865482 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2216
I20260929 02:46:10.874399 2788 tablet_replica.cc:330] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: stopping tablet replica
I20260929 02:46:10.874676 2788 raft_consensus.cc:2229] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:10.874902 2789 tablet_replica.cc:330] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: stopping tablet replica
I20260929 02:46:10.874962 2788 raft_consensus.cc:2258] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:10.875258 2790 tablet_replica.cc:330] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: stopping tablet replica
I20260929 02:46:10.875440 2789 raft_consensus.cc:2229] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:10.875519 2790 raft_consensus.cc:2229] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:10.875834 2790 raft_consensus.cc:2258] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:10.875900 2789 raft_consensus.cc:2258] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:10.875926 2788 ts_tablet_manager.cc:1905] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:10.876899 2789 ts_tablet_manager.cc:1905] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:10.877377 2790 ts_tablet_manager.cc:1905] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:10.881354 2788 ts_tablet_manager.cc:1918] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:46:10.881513 2788 log.cc:1196] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/5c2df21eb0ab467690404bfcec9b4a3d
I20260929 02:46:10.882041 2788 ts_tablet_manager.cc:1939] T 5c2df21eb0ab467690404bfcec9b4a3d P 763f17acab1941d792bfa92beeb2162d: Deleting consensus metadata
I20260929 02:46:10.883085 2132 catalog_manager.cc:4841] TS 763f17acab1941d792bfa92beeb2162d (127.0.97.1:42441): tablet 5c2df21eb0ab467690404bfcec9b4a3d (table test-table [id=4eb19502dbac452383a72c3515c089e3]) successfully deleted
I20260929 02:46:10.883417 2789 ts_tablet_manager.cc:1918] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:46:10.883538 2789 log.cc:1196] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/5c2df21eb0ab467690404bfcec9b4a3d
I20260929 02:46:10.883955 2789 ts_tablet_manager.cc:1939] T 5c2df21eb0ab467690404bfcec9b4a3d P 1f003bde72b74706a5fffad865915b35: Deleting consensus metadata
I20260929 02:46:10.884320 2790 ts_tablet_manager.cc:1918] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 02:46:10.884449 2790 log.cc:1196] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/5c2df21eb0ab467690404bfcec9b4a3d
I20260929 02:46:10.884938 2790 ts_tablet_manager.cc:1939] T 5c2df21eb0ab467690404bfcec9b4a3d P a4edb27278484b4cb4e7945424cfaf37: Deleting consensus metadata
I20260929 02:46:10.885162 2132 catalog_manager.cc:4841] TS 1f003bde72b74706a5fffad865915b35 (127.0.97.2:39825): tablet 5c2df21eb0ab467690404bfcec9b4a3d (table test-table [id=4eb19502dbac452383a72c3515c089e3]) successfully deleted
I20260929 02:46:10.886073 2130 catalog_manager.cc:4841] TS a4edb27278484b4cb4e7945424cfaf37 (127.0.97.3:41519): tablet 5c2df21eb0ab467690404bfcec9b4a3d (table test-table [id=4eb19502dbac452383a72c3515c089e3]) successfully deleted
Sep 29 02:46:10 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:10.895147 2792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:10.867108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.0.97.1:42441)
0929 02:46:10.869101 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:10.869111 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:10.869139 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:46:10.869207 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:10.869215 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:10.869312 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:10.871514 (+ 2202us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:10.871768 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:10.873849 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:10.876679 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:10.877344 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:10.877464 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:10.888905 (+ 11441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:10.888920 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:10.888936 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:10.888973 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:10.892309 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:10.893075 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:10.893083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:10.893086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:10.893230 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:10.893635 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:10.893639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:10.893641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:10.893961 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:10.894296 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:10.894577 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 02:46:10.894819 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1799,"thread_start_us":1635,"threads_started":1}
W20260929 02:46:10.895857 2347 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47982
W20260929 02:46:10.896462 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2216: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:10.896739 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 2216
I20260929 02:46:10.921936 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2356
Sep 29 02:46:10 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:46:10.937911 2793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:10.922698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.0.97.2:39825)
0929 02:46:10.923269 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:10.923274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:10.923290 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:46:10.923343 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:10.923346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:10.923401 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:10.923498 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:10.924835 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:10.925595 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:10.927238 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:10.927899 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:10.928038 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:10.932485 (+ 4447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:10.932498 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:10.932515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:10.932594 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:10.935455 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:10.936058 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:10.936065 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:10.936068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:10.936192 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:10.936493 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:10.936496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:10.936498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:10.936929 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:10.937140 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:10.937389 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 02:46:10.937544 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":353,"threads_started":1}
W20260929 02:46:10.938630 2486 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59146
W20260929 02:46:10.939256 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2356: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:10.939419 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 2356
I20260929 02:46:10.961062 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2495
Sep 29 02:46:10 dist-test-slave-6tjh krb5kdc[2100](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649968, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:46:10.975498 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:10.961792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.0.97.3:41519)
0929 02:46:10.962296 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:10.962301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:10.962316 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:46:10.962368 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:10.962371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:10.962426 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:10.962523 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:10.963742 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:10.964436 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:10.965686 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:10.966090 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:10.966194 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:10.970433 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:10.970447 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:10.970463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:10.970493 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:10.973343 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:10.973902 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:10.973909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:10.973913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:10.974038 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:10.974337 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:10.974340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:10.974342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:10.974654 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:10.974884 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:10.975082 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 02:46:10.975237 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
W20260929 02:46:10.976199 2625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37180
W20260929 02:46:10.976818 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2495: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:10.976994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 2495
I20260929 02:46:10.993892 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2113
I20260929 02:46:10.995042 2185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:11.112517 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 2113
2026-09-29T02:46:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8270 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:11 dist-test-slave-6tjh krb5kdc[2798](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:11 dist-test-slave-6tjh krb5kdc[2798](info): set up 2 sockets
Sep 29 02:46:11 dist-test-slave-6tjh krb5kdc[2798](info): commencing operation
krb5kdc: starting...
W20260929 02:46:12.196808 388 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:12 dist-test-slave-6tjh krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:12.389271 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38783
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38783
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:12.703877 2811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:12.704183 2811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:12.704346 2811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:12.711740 2811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:12.711913 2811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:12.711995 2811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:12.712075 2811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:12.712141 2811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:12.722173 2811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:12.723647 2811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:12.725092 2811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:12.737280 2816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:12.737196 2817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:12.737210 2819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:12.738368 2811 server_base.cc:1034] running on GCE node
I20260929 02:46:12.739269 2811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:12.740825 2811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:12.742059 2811 hybrid_clock.cc:648] HybridClock initialized: now 1790649972742022 us; error 53 us; skew 500 ppm
I20260929 02:46:12.742406 2811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:12 dist-test-slave-6tjh krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:12.745477 2811 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:12.746953 2811 webserver.cc:460] Webserver started at http://127.0.97.62:33627/ using document root <none> and password file <none>
I20260929 02:46:12.747648 2811 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:12.747802 2811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:12.748121 2811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:12.751204 2811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "ee16ae0339e04580af51fe49c2c2e234"
format_stamp: "Formatted at 2026-09-29 02:46:12 on dist-test-slave-6tjh"
server_key: "026d58802d20a91df3cf781539b6bab5"
server_key_iv: "f0877aaa94d02f64f87b78fb1ba0e1e5"
server_key_version: "encryptionkey@0"
I20260929 02:46:12.752009 2811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "ee16ae0339e04580af51fe49c2c2e234"
format_stamp: "Formatted at 2026-09-29 02:46:12 on dist-test-slave-6tjh"
server_key: "026d58802d20a91df3cf781539b6bab5"
server_key_iv: "f0877aaa94d02f64f87b78fb1ba0e1e5"
server_key_version: "encryptionkey@0"
I20260929 02:46:12.757648 2811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:46:12.761328 2826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:12.762840 2811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:12.763052 2811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "ee16ae0339e04580af51fe49c2c2e234"
format_stamp: "Formatted at 2026-09-29 02:46:12 on dist-test-slave-6tjh"
server_key: "026d58802d20a91df3cf781539b6bab5"
server_key_iv: "f0877aaa94d02f64f87b78fb1ba0e1e5"
server_key_version: "encryptionkey@0"
I20260929 02:46:12.763262 2811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:12.776834 2811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:12.781054 2811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:12.781359 2811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:12.799227 2811 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38783
I20260929 02:46:12.799250 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38783 every 8 connection(s)
I20260929 02:46:12.800686 2811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:12.804458 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:12.807225 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 2811
I20260929 02:46:12.807462 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:12.808211 388 external_mini_cluster.cc:1428] Setting key 284772aa070a8337d9e5523f139c909f
I20260929 02:46:12.817637 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234: Bootstrap starting.
I20260929 02:46:12.821625 2878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:12.823050 2878 log.cc:824] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:46:12 dist-test-slave-6tjh krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:12.826125 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234: No bootstrap required, opened a new log
I20260929 02:46:12.832127 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:12.810418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.0.97.62:38783)
0929 02:46:12.811221 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:12.811233 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:12.811275 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:46:12.813116 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:12.813140 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:12.813566 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:12.813928 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:12.815785 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:12.817364 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:12.819173 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:12.819924 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:12.820063 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:12.824997 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:12.825005 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:12.825020 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:12.825053 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:12.828106 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:12.828913 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:12.828919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:12.828925 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:12.829086 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:12.829480 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:12.829483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:12.829485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:12.830015 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:12.830228 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:12.830654 (+ 426us) server_negotiation.cc:299] Negotiation successful
0929 02:46:12.831065 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":394,"threads_started":1}
I20260929 02:46:12.835192 2878 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee16ae0339e04580af51fe49c2c2e234" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38783 } }
I20260929 02:46:12.835568 2878 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:12.835672 2878 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee16ae0339e04580af51fe49c2c2e234, State: Initialized, Role: FOLLOWER
I20260929 02:46:12.836199 2878 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee16ae0339e04580af51fe49c2c2e234" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38783 } }
I20260929 02:46:12.836438 2878 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:12.836611 2878 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:12.836787 2878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:12.838855 2878 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee16ae0339e04580af51fe49c2c2e234" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38783 } }
I20260929 02:46:12.839322 2878 leader_election.cc:302] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee16ae0339e04580af51fe49c2c2e234; no voters:
I20260929 02:46:12.839915 2878 leader_election.cc:288] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:12.839995 2883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:12.841545 2883 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [term 1 LEADER]: Becoming Leader. State: Replica: ee16ae0339e04580af51fe49c2c2e234, State: Running, Role: LEADER
I20260929 02:46:12.842159 2878 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:12.842099 2883 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee16ae0339e04580af51fe49c2c2e234" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38783 } }
I20260929 02:46:12.845150 2885 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee16ae0339e04580af51fe49c2c2e234. Latest consensus state: current_term: 1 leader_uuid: "ee16ae0339e04580af51fe49c2c2e234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee16ae0339e04580af51fe49c2c2e234" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38783 } } }
I20260929 02:46:12.845562 2885 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:12.846102 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee16ae0339e04580af51fe49c2c2e234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee16ae0339e04580af51fe49c2c2e234" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38783 } } }
I20260929 02:46:12.846396 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:12.849531 2893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:12.853960 2893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:12.862764 2893 catalog_manager.cc:1347] Generated new cluster ID: 91f6f8c9b62b410d8c8e5e02ee74b12d
I20260929 02:46:12.862901 2893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:12.870620 2893 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:12.871935 2893 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:12.884699 2893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee16ae0339e04580af51fe49c2c2e234: Generated new TSK 0
I20260929 02:46:12.885952 2893 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:46:12.936640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38783
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:13.327113 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:13.327410 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:13.327505 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:13.335422 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:13.335546 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:13.335664 2904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:13.345979 2904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:13.347255 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:13.348748 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:13.363011 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:13.362991 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:13.362964 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:13.363849 2904 server_base.cc:1034] running on GCE node
I20260929 02:46:13.364483 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:13.365506 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:13.366724 2904 hybrid_clock.cc:648] HybridClock initialized: now 1790649973366655 us; error 43 us; skew 500 ppm
I20260929 02:46:13.367107 2904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:13 dist-test-slave-6tjh krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:13.370824 2904 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:46:13.372516 2904 webserver.cc:460] Webserver started at http://127.0.97.1:37329/ using document root <none> and password file <none>
I20260929 02:46:13.373450 2904 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:13.373641 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:13.374015 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:13.377527 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "3ca50b77a4c74df49d7d720df0afd78c"
format_stamp: "Formatted at 2026-09-29 02:46:13 on dist-test-slave-6tjh"
server_key: "c80778c5bc910b47293010d0aa602e89"
server_key_iv: "2d38bd1fc1a0ffcb9a5fde56fedf0cf5"
server_key_version: "encryptionkey@0"
I20260929 02:46:13.378414 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3ca50b77a4c74df49d7d720df0afd78c"
format_stamp: "Formatted at 2026-09-29 02:46:13 on dist-test-slave-6tjh"
server_key: "c80778c5bc910b47293010d0aa602e89"
server_key_iv: "2d38bd1fc1a0ffcb9a5fde56fedf0cf5"
server_key_version: "encryptionkey@0"
I20260929 02:46:13.384357 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:46:13.388314 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:13.389976 2904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:13.390177 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "3ca50b77a4c74df49d7d720df0afd78c"
format_stamp: "Formatted at 2026-09-29 02:46:13 on dist-test-slave-6tjh"
server_key: "c80778c5bc910b47293010d0aa602e89"
server_key_iv: "2d38bd1fc1a0ffcb9a5fde56fedf0cf5"
server_key_version: "encryptionkey@0"
I20260929 02:46:13.390407 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:13.411610 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:13.416435 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:13.416795 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:13.417869 2904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:13.419309 2904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:13.419417 2904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:13.419576 2904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:13.419648 2904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:13.453732 2904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42079
I20260929 02:46:13.453842 3031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42079 every 8 connection(s)
I20260929 02:46:13.455128 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:13.459290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 2904
I20260929 02:46:13.459537 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:13.460362 388 external_mini_cluster.cc:1428] Setting key e22d52ef96bb216d031a3afa804a04a3
Sep 29 02:46:13 dist-test-slave-6tjh krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:13.480521 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:13.457854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48333 (local address 127.0.97.62:38783)
0929 02:46:13.458312 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:13.458316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:13.459385 (+ 1069us) server_negotiation.cc:407] Connection header received
0929 02:46:13.461952 (+ 2567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:13.461958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:13.462031 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:13.462156 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:13.464391 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.465311 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.468025 (+ 2714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.468843 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.469016 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:13.472539 (+ 3523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:13.472577 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:13.472584 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:13.472635 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:13.476612 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:13.477831 (+ 1219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:13.477836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:13.477838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:13.477968 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:13.478594 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:13.478598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:13.478600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:13.478881 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:13.478995 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:13.480088 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0929 02:46:13.480267 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":253,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:46:13.482323 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:13.458302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38783 (local address 127.0.97.1:48333)
0929 02:46:13.459138 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:13.459193 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:13.461663 (+ 2470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:13.462469 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:13.462479 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:13.463259 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:13.464149 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:13.464179 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.465499 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.465508 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:13.467800 (+ 2292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:13.467819 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.469083 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.469088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:13.469323 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:13.470121 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:13.470148 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:13.472292 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:13.476856 (+ 4564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:13.476869 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:13.476894 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:13.477653 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:13.478118 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:13.478127 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:13.478135 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:13.478442 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:13.479043 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:13.479053 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:13.479597 (+ 544us) client_negotiation.cc:769] Sending connection context
0929 02:46:13.479993 (+ 396us) client_negotiation.cc:240] Negotiation successful
0929 02:46:13.480682 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":265,"threads_started":1}
I20260929 02:46:13.484061 3032 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38783
I20260929 02:46:13.484365 3032 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:13.485273 3032 heartbeater.cc:511] Master 127.0.97.62:38783 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:46:13.487536 2843 ts_manager.cc:194] Registered new tserver with Master: 3ca50b77a4c74df49d7d720df0afd78c (127.0.97.1:42079)
I20260929 02:46:13.489126 2843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:13.509343 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38783
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:13.838974 3038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:13.839270 3038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:13.839435 3038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:13.846920 3038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:13.847044 3038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:13.847225 3038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:13.857621 3038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:13.858978 3038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:13.860401 3038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:13.874264 3046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:13.874282 3044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:13.874361 3043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:13.875137 3038 server_base.cc:1034] running on GCE node
I20260929 02:46:13.875761 3038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:13.876899 3038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:13.878165 3038 hybrid_clock.cc:648] HybridClock initialized: now 1790649973878089 us; error 74 us; skew 500 ppm
I20260929 02:46:13.878561 3038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:13 dist-test-slave-6tjh krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:13.882225 3038 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:13.883765 3038 webserver.cc:460] Webserver started at http://127.0.97.2:41191/ using document root <none> and password file <none>
I20260929 02:46:13.884500 3038 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:13.884737 3038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:13.885135 3038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:13.888315 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "39aeb72c9cba487380bf376999db818a"
format_stamp: "Formatted at 2026-09-29 02:46:13 on dist-test-slave-6tjh"
server_key: "fe43449dbcfc387d4ed858b6d9a4f10a"
server_key_iv: "efd583972678467109eac0c03c69179d"
server_key_version: "encryptionkey@0"
I20260929 02:46:13.889184 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "39aeb72c9cba487380bf376999db818a"
format_stamp: "Formatted at 2026-09-29 02:46:13 on dist-test-slave-6tjh"
server_key: "fe43449dbcfc387d4ed858b6d9a4f10a"
server_key_iv: "efd583972678467109eac0c03c69179d"
server_key_version: "encryptionkey@0"
I20260929 02:46:13.894774 3038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:46:13.898819 3053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:13.900421 3038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:13.900653 3038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "39aeb72c9cba487380bf376999db818a"
format_stamp: "Formatted at 2026-09-29 02:46:13 on dist-test-slave-6tjh"
server_key: "fe43449dbcfc387d4ed858b6d9a4f10a"
server_key_iv: "efd583972678467109eac0c03c69179d"
server_key_version: "encryptionkey@0"
I20260929 02:46:13.900873 3038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:13.917584 3038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:13.922204 3038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:13.922449 3038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:13.923403 3038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:13.924811 3038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:13.924885 3038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:13.924971 3038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:13.925021 3038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:13.960256 3038 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36347
I20260929 02:46:13.960299 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36347 every 8 connection(s)
I20260929 02:46:13.961625 3038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:13.968338 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3038
I20260929 02:46:13.968561 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:13.969110 388 external_mini_cluster.cc:1428] Setting key d4696eb796d6125764f2729cf38edb20
Sep 29 02:46:13 dist-test-slave-6tjh krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:13.983045 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:13.964422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40451 (local address 127.0.97.62:38783)
0929 02:46:13.964716 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:13.964721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:13.965894 (+ 1173us) server_negotiation.cc:407] Connection header received
0929 02:46:13.967862 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:13.967867 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:13.967978 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:13.968075 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:13.970112 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.970940 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.972573 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.973369 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.973512 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:13.977212 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:13.977217 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:13.977220 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:13.977250 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:13.979677 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:13.980638 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:13.980641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:13.980643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:13.980761 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:13.981261 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:13.981264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:13.981265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:13.981499 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:13.981604 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:13.982528 (+ 924us) server_negotiation.cc:299] Negotiation successful
0929 02:46:13.982700 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:46:13.984686 3168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:13.964981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38783 (local address 127.0.97.2:40451)
0929 02:46:13.965717 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:13.965764 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:13.967567 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:13.968382 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:13.968395 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:13.969190 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:13.969961 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:13.969980 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.971105 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.971114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:13.972394 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:13.972406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:13.973626 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:13.973632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:13.973865 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:13.974619 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:13.974642 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:13.976998 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:13.979813 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:13.979825 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:13.979845 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:13.980481 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:13.980892 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:13.980899 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:13.980905 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:13.981154 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:13.981624 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:13.981635 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:13.982109 (+ 474us) client_negotiation.cc:769] Sending connection context
0929 02:46:13.982437 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 02:46:13.983095 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":291,"threads_started":1}
I20260929 02:46:13.986294 3166 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38783
I20260929 02:46:13.986630 3166 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:13.987428 3166 heartbeater.cc:511] Master 127.0.97.62:38783 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:46:13.988986 2843 ts_manager.cc:194] Registered new tserver with Master: 39aeb72c9cba487380bf376999db818a (127.0.97.2:36347)
I20260929 02:46:13.989893 2843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40451
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:14.015669 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38783
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:37615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:14.352962 3171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:14.353259 3171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:14.353415 3171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:14.360965 3171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:14.361086 3171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:14.361263 3171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:14.372174 3171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:37615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:14.373678 3171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:14.375099 3171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:14.388935 3179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:14.388893 3177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:14.389611 3171 server_base.cc:1034] running on GCE node
W20260929 02:46:14.389632 3176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:14.390341 3171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:14.391305 3171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:14.392536 3171 hybrid_clock.cc:648] HybridClock initialized: now 1790649974392485 us; error 53 us; skew 500 ppm
I20260929 02:46:14.392904 3171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:14 dist-test-slave-6tjh krb5kdc[2798](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:14.396142 3171 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:14.397634 3171 webserver.cc:460] Webserver started at http://127.0.97.3:36585/ using document root <none> and password file <none>
I20260929 02:46:14.398355 3171 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:14.398501 3171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:14.398821 3171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:14.401831 3171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "77153a1c936d4d8aa0be52c975c65968"
format_stamp: "Formatted at 2026-09-29 02:46:14 on dist-test-slave-6tjh"
server_key: "d69f682f18daf25e919fdbb4344fc7ae"
server_key_iv: "92fd4e20bf60df3dc8a70637eeef121b"
server_key_version: "encryptionkey@0"
I20260929 02:46:14.402611 3171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "77153a1c936d4d8aa0be52c975c65968"
format_stamp: "Formatted at 2026-09-29 02:46:14 on dist-test-slave-6tjh"
server_key: "d69f682f18daf25e919fdbb4344fc7ae"
server_key_iv: "92fd4e20bf60df3dc8a70637eeef121b"
server_key_version: "encryptionkey@0"
I20260929 02:46:14.408119 3171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:46:14.411664 3186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:14.413084 3171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:14.413259 3171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "77153a1c936d4d8aa0be52c975c65968"
format_stamp: "Formatted at 2026-09-29 02:46:14 on dist-test-slave-6tjh"
server_key: "d69f682f18daf25e919fdbb4344fc7ae"
server_key_iv: "92fd4e20bf60df3dc8a70637eeef121b"
server_key_version: "encryptionkey@0"
I20260929 02:46:14.413456 3171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:14.434245 3171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:14.438733 3171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:14.439016 3171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:14.439945 3171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:14.441309 3171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:14.441386 3171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:14.441468 3171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:14.441509 3171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:14.475416 3171 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33771
I20260929 02:46:14.475543 3298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33771 every 8 connection(s)
I20260929 02:46:14.476783 3171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:14.485327 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3171
I20260929 02:46:14.485585 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:14.486143 388 external_mini_cluster.cc:1428] Setting key fcb5420532f0d874bbb5f19e1e65ed84
Sep 29 02:46:14 dist-test-slave-6tjh krb5kdc[2798](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:14.492242 3032 heartbeater.cc:503] Master 127.0.97.62:38783 was elected leader, sending a full tablet report...
I20260929 02:46:14.498796 3034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:14.479512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34709 (local address 127.0.97.62:38783)
0929 02:46:14.479726 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:14.479731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:14.481014 (+ 1283us) server_negotiation.cc:407] Connection header received
0929 02:46:14.483042 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:14.483048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:14.483149 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:14.483283 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:14.485015 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:14.485984 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:14.487720 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:14.488173 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:14.488328 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:14.492129 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:14.492135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:14.492139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:14.492173 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:14.495476 (+ 3303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:14.496284 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:14.496289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:14.496291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:14.496417 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:14.496907 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:14.496917 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:14.496920 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:14.497194 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:14.497368 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:14.498165 (+ 797us) server_negotiation.cc:299] Negotiation successful
0929 02:46:14.498433 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:46:14.499861 3301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:14.479991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38783 (local address 127.0.97.3:34709)
0929 02:46:14.480838 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:14.480880 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:14.482740 (+ 1860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:14.483531 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:14.483540 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:14.484102 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:14.484814 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:14.484833 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:14.486143 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:14.486157 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:14.487551 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:14.487564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:14.488441 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:14.488448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:14.488781 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:14.489550 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:14.489580 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:14.491892 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:14.495632 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:14.495642 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:14.495666 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:14.496131 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:14.496536 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:14.496542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:14.496572 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:14.496775 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:14.497341 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:14.497350 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:14.497729 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 02:46:14.498043 (+ 314us) client_negotiation.cc:240] Negotiation successful
0929 02:46:14.498613 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":285,"threads_started":1}
I20260929 02:46:14.501569 3299 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38783
I20260929 02:46:14.501892 3299 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:14.502609 3299 heartbeater.cc:511] Master 127.0.97.62:38783 requested a full tablet report, sending...
I20260929 02:46:14.503989 2843 ts_manager.cc:194] Registered new tserver with Master: 77153a1c936d4d8aa0be52c975c65968 (127.0.97.3:33771)
I20260929 02:46:14.504941 2843 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:34709
I20260929 02:46:14.514447 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:46:14.545944 3034 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:14.542943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53634 (local address 127.0.97.62:38783)
0929 02:46:14.543214 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:14.543223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:14.543576 (+ 353us) server_negotiation.cc:407] Connection header received
0929 02:46:14.543860 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:14.543867 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:14.543951 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:14.544077 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:14.545631 (+ 1554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53634: BlockingRecv error: recv got EOF from 127.0.0.1:53634 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 02:46:14.549160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2904
W20260929 02:46:14.552115 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2904: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:42079: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:46:14.552280 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 2904
I20260929 02:46:14.567778 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3038
W20260929 02:46:14.570478 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3038: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:36347: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:46:14.570659 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3038
I20260929 02:46:14.585556 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3171
W20260929 02:46:14.588517 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3171: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:33771: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:46:14.588711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3171
I20260929 02:46:14.603627 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 2811
I20260929 02:46:14.604993 2873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:14.708382 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 2811
2026-09-29T02:46:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3594 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:14 dist-test-slave-6tjh krb5kdc[3316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:14 dist-test-slave-6tjh krb5kdc[3316](info): set up 2 sockets
Sep 29 02:46:14 dist-test-slave-6tjh krb5kdc[3316](info): commencing operation
krb5kdc: starting...
W20260929 02:46:15.801029 388 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:15 dist-test-slave-6tjh krb5kdc[3316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:15.997162 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33797
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:16.319525 3329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:16.319878 3329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:16.320055 3329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:16.327595 3329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:16.327751 3329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:16.327875 3329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:16.327970 3329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:16.328061 3329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:16.338173 3329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:16.339515 3329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:16.341002 3329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:16.353233 3334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:16.353331 3335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:16.353351 3337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:16.353638 3329 server_base.cc:1034] running on GCE node
I20260929 02:46:16.354694 3329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:16.356082 3329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:16.357307 3329 hybrid_clock.cc:648] HybridClock initialized: now 1790649976357270 us; error 39 us; skew 500 ppm
I20260929 02:46:16.357677 3329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:16 dist-test-slave-6tjh krb5kdc[3316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:16.366990 3329 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:16.368695 3329 webserver.cc:460] Webserver started at http://127.0.97.62:46337/ using document root <none> and password file <none>
I20260929 02:46:16.369439 3329 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:16.369591 3329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:16.369913 3329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:16.373054 3329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "8a7ba036190a4374954546a1143ef031"
format_stamp: "Formatted at 2026-09-29 02:46:16 on dist-test-slave-6tjh"
server_key: "2d3510614bc9f3057b307bdebd0d185a"
server_key_iv: "a20cdffb259c36a590986848fdc5c5c0"
server_key_version: "encryptionkey@0"
I20260929 02:46:16.373847 3329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a7ba036190a4374954546a1143ef031"
format_stamp: "Formatted at 2026-09-29 02:46:16 on dist-test-slave-6tjh"
server_key: "2d3510614bc9f3057b307bdebd0d185a"
server_key_iv: "a20cdffb259c36a590986848fdc5c5c0"
server_key_version: "encryptionkey@0"
I20260929 02:46:16.378909 3329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 02:46:16.382249 3344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:16.383500 3329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:46:16.383674 3329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "8a7ba036190a4374954546a1143ef031"
format_stamp: "Formatted at 2026-09-29 02:46:16 on dist-test-slave-6tjh"
server_key: "2d3510614bc9f3057b307bdebd0d185a"
server_key_iv: "a20cdffb259c36a590986848fdc5c5c0"
server_key_version: "encryptionkey@0"
I20260929 02:46:16.383872 3329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:16.403173 3329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:16.407466 3329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:16.407768 3329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:16.425966 3329 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33797
I20260929 02:46:16.425988 3395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33797 every 8 connection(s)
I20260929 02:46:16.427452 3329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:16.431491 3396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:16.435500 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3329
I20260929 02:46:16.435716 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:16.436355 388 external_mini_cluster.cc:1428] Setting key 071f3a4b61e3d92f511a51f497273270
I20260929 02:46:16.442842 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031: Bootstrap starting.
I20260929 02:46:16.446529 3396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:16.447665 3396 log.cc:824] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:16.451145 3396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031: No bootstrap required, opened a new log
Sep 29 02:46:16 dist-test-slave-6tjh krb5kdc[3316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:16.458966 3396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ba036190a4374954546a1143ef031" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:46:16.459466 3396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:16.459576 3396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7ba036190a4374954546a1143ef031, State: Initialized, Role: FOLLOWER
I20260929 02:46:16.460307 3396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ba036190a4374954546a1143ef031" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:46:16.460602 3396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:16.460649 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:16.438330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48272 (local address 127.0.97.62:33797)
0929 02:46:16.439128 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:16.439141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:16.439189 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:46:16.441190 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:16.441213 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:16.441623 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:16.441970 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:16.443815 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:16.445234 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:16.447031 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:16.447685 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:16.447872 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:16.452539 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:16.452600 (+ 61us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:16.452626 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:16.452669 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:16.455910 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:16.456714 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:16.456722 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:16.456729 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:16.456892 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:16.457293 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:16.457301 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:16.457304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:16.458055 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:16.458349 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:16.458811 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 02:46:16.459256 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":375,"threads_started":1}
I20260929 02:46:16.460763 3396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:16.460949 3396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:16.463977 3396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ba036190a4374954546a1143ef031" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:46:16.464622 3396 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7ba036190a4374954546a1143ef031; no voters:
I20260929 02:46:16.464840 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:46:16.465258 3396 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:16.465404 3401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:16.467235 3396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:16.474150 3401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7ba036190a4374954546a1143ef031, State: Running, Role: LEADER
I20260929 02:46:16.474836 3401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ba036190a4374954546a1143ef031" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:46:16.478361 3402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7ba036190a4374954546a1143ef031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ba036190a4374954546a1143ef031" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } } }
I20260929 02:46:16.478719 3402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:16.479241 3403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7ba036190a4374954546a1143ef031. Latest consensus state: current_term: 1 leader_uuid: "8a7ba036190a4374954546a1143ef031" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7ba036190a4374954546a1143ef031" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } } }
I20260929 02:46:16.479523 3403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031 [sys.catalog]: This master's current role is: LEADER
W20260929 02:46:16.494127 3416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:46:16.494275 3416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:46:16.494416 3417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:16.496632 3417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:16.506146 3417 catalog_manager.cc:1347] Generated new cluster ID: bf04e6977b7342298507ae287f274cf1
I20260929 02:46:16.506382 3417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:16.518087 3399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:16.508695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48282 (local address 127.0.97.62:33797)
0929 02:46:16.508879 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:16.508885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:16.509490 (+ 605us) server_negotiation.cc:407] Connection header received
0929 02:46:16.509692 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:16.509698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:16.509772 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:16.509877 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:16.511165 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:16.512332 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:16.514314 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:16.514969 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:16.515092 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:16.516348 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:16.516354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:16.516359 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:16.517239 (+ 880us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:16.517304 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:16.517326 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:16.517454 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:16.517592 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 02:46:16.517688 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":63}
I20260929 02:46:16.518723 3417 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:16.520267 3417 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:16.535008 3417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7ba036190a4374954546a1143ef031: Generated new TSK 0
I20260929 02:46:16.536221 3417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:46:16.547741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3329
I20260929 02:46:16.548981 3391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:16.638590 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3329
2026-09-29T02:46:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1925 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:16 dist-test-slave-6tjh krb5kdc[3430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:16 dist-test-slave-6tjh krb5kdc[3430](info): set up 2 sockets
Sep 29 02:46:16 dist-test-slave-6tjh krb5kdc[3430](info): commencing operation
krb5kdc: starting...
W20260929 02:46:17.720947 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.010s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:17 dist-test-slave-6tjh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:17.921077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43339
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:18.245437 3443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:18.245903 3443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:18.246096 3443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:18.254784 3443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:18.255007 3443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:18.255128 3443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:18.255245 3443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:18.255342 3443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:18.268038 3443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:18.269591 3443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:18.271054 3443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:18.283160 3448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:18.283107 3449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:18.283183 3451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:18.283944 3443 server_base.cc:1034] running on GCE node
I20260929 02:46:18.284821 3443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:18.286226 3443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:18.287472 3443 hybrid_clock.cc:648] HybridClock initialized: now 1790649978287420 us; error 56 us; skew 500 ppm
I20260929 02:46:18.287818 3443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:18 dist-test-slave-6tjh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:18.291002 3443 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:18.292439 3443 webserver.cc:460] Webserver started at http://127.0.97.62:46587/ using document root <none> and password file <none>
I20260929 02:46:18.293156 3443 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:18.293303 3443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:18.293604 3443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:18.296509 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "837b9026670b4c9a895846390319f63e"
format_stamp: "Formatted at 2026-09-29 02:46:18 on dist-test-slave-6tjh"
server_key: "c21e55b839699dbb26789b8d5b8922f4"
server_key_iv: "66bb98f2d75a2068b53e2bb21f9c1774"
server_key_version: "encryptionkey@0"
I20260929 02:46:18.297338 3443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "837b9026670b4c9a895846390319f63e"
format_stamp: "Formatted at 2026-09-29 02:46:18 on dist-test-slave-6tjh"
server_key: "c21e55b839699dbb26789b8d5b8922f4"
server_key_iv: "66bb98f2d75a2068b53e2bb21f9c1774"
server_key_version: "encryptionkey@0"
I20260929 02:46:18.302500 3443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:46:18.305835 3458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:18.307094 3443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:18.307276 3443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "837b9026670b4c9a895846390319f63e"
format_stamp: "Formatted at 2026-09-29 02:46:18 on dist-test-slave-6tjh"
server_key: "c21e55b839699dbb26789b8d5b8922f4"
server_key_iv: "66bb98f2d75a2068b53e2bb21f9c1774"
server_key_version: "encryptionkey@0"
I20260929 02:46:18.307471 3443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:18.321377 3443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:18.325608 3443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:18.325910 3443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:18.344579 3443 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43339
I20260929 02:46:18.344624 3509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43339 every 8 connection(s)
I20260929 02:46:18.345980 3443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:18.349641 3510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:18.349684 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3443
I20260929 02:46:18.349901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:18.350541 388 external_mini_cluster.cc:1428] Setting key e8347f921343b7910c52b1a771a308de
I20260929 02:46:18.361832 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e: Bootstrap starting.
Sep 29 02:46:18 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649977, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:18.365182 3510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:18.366365 3510 log.cc:824] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:18.369571 3510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e: No bootstrap required, opened a new log
I20260929 02:46:18.373423 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:18.352577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.0.97.62:43339)
0929 02:46:18.353378 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:18.353390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:18.353429 (+ 39us) server_negotiation.cc:407] Connection header received
0929 02:46:18.355601 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:18.355624 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:18.356056 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:18.356472 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:18.357906 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:18.359255 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:18.360538 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:18.361239 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:18.361437 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:18.365561 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:18.365572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:18.365586 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:18.365635 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:18.369045 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:18.369812 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:18.369820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:18.369826 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:18.369995 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:18.370472 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:18.370475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:18.370477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:18.370995 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:18.371218 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:18.371757 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 02:46:18.372249 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":386,"threads_started":1}
I20260929 02:46:18.377825 3510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837b9026670b4c9a895846390319f63e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43339 } }
I20260929 02:46:18.378240 3510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:18.378356 3510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 837b9026670b4c9a895846390319f63e, State: Initialized, Role: FOLLOWER
I20260929 02:46:18.378921 3510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837b9026670b4c9a895846390319f63e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43339 } }
I20260929 02:46:18.379159 3510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:18.379308 3510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:18.379490 3510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:18.381744 3510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837b9026670b4c9a895846390319f63e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43339 } }
I20260929 02:46:18.382267 3510 leader_election.cc:302] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 837b9026670b4c9a895846390319f63e; no voters:
I20260929 02:46:18.382899 3510 leader_election.cc:288] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:18.383059 3515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:18.383848 3515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [term 1 LEADER]: Becoming Leader. State: Replica: 837b9026670b4c9a895846390319f63e, State: Running, Role: LEADER
I20260929 02:46:18.384407 3515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837b9026670b4c9a895846390319f63e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43339 } }
I20260929 02:46:18.385159 3510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:18.387754 3516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "837b9026670b4c9a895846390319f63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837b9026670b4c9a895846390319f63e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43339 } } }
I20260929 02:46:18.387921 3517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 837b9026670b4c9a895846390319f63e. Latest consensus state: current_term: 1 leader_uuid: "837b9026670b4c9a895846390319f63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "837b9026670b4c9a895846390319f63e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43339 } } }
I20260929 02:46:18.388317 3516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:18.388367 3517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:18.390126 3523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:18.395596 3523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:18.404906 3523 catalog_manager.cc:1347] Generated new cluster ID: 0e2f5b9f9c82443e9255dc3e4ba6b107
I20260929 02:46:18.405038 3523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:18.412232 3523 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:18.414141 3523 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:18.432677 3523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 837b9026670b4c9a895846390319f63e: Generated new TSK 0
I20260929 02:46:18.433702 3523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:46:18.496658 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43339
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:18.830895 3536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:18.831179 3536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:18.831333 3536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:18.838907 3536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:18.839077 3536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:18.839290 3536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:18.849414 3536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:18.850800 3536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:18.852164 3536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:18.866165 3541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:18.866209 3544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:18.866247 3542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:18.867255 3536 server_base.cc:1034] running on GCE node
I20260929 02:46:18.867939 3536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:18.869000 3536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:18.870263 3536 hybrid_clock.cc:648] HybridClock initialized: now 1790649978870225 us; error 45 us; skew 500 ppm
I20260929 02:46:18.870609 3536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:18 dist-test-slave-6tjh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:18.874369 3536 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:46:18.875865 3536 webserver.cc:460] Webserver started at http://127.0.97.1:42825/ using document root <none> and password file <none>
I20260929 02:46:18.876509 3536 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:18.876679 3536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:18.876936 3536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:18.879881 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "9831663acd02411f86f42f4a0ae2a1ab"
format_stamp: "Formatted at 2026-09-29 02:46:18 on dist-test-slave-6tjh"
server_key: "6051a8a487967da32dfe97e25ed5f7c6"
server_key_iv: "ffe5ec6d59102c4f4a7b0b57cdb2d136"
server_key_version: "encryptionkey@0"
I20260929 02:46:18.880704 3536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9831663acd02411f86f42f4a0ae2a1ab"
format_stamp: "Formatted at 2026-09-29 02:46:18 on dist-test-slave-6tjh"
server_key: "6051a8a487967da32dfe97e25ed5f7c6"
server_key_iv: "ffe5ec6d59102c4f4a7b0b57cdb2d136"
server_key_version: "encryptionkey@0"
I20260929 02:46:18.885710 3536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:18.889119 3551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:18.890728 3536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:46:18.890913 3536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "9831663acd02411f86f42f4a0ae2a1ab"
format_stamp: "Formatted at 2026-09-29 02:46:18 on dist-test-slave-6tjh"
server_key: "6051a8a487967da32dfe97e25ed5f7c6"
server_key_iv: "ffe5ec6d59102c4f4a7b0b57cdb2d136"
server_key_version: "encryptionkey@0"
I20260929 02:46:18.891080 3536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:18.917083 3536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:18.921494 3536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:18.921808 3536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:18.922783 3536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:18.924124 3536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:18.924206 3536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:18.924326 3536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:18.924391 3536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:18.957718 3536 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35863
I20260929 02:46:18.957759 3663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35863 every 8 connection(s)
I20260929 02:46:18.959012 3536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:18.966020 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3536
I20260929 02:46:18.966274 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:18.966945 388 external_mini_cluster.cc:1428] Setting key 4a7b828eadbc578907d4bdc874ffddec
Sep 29 02:46:18 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:18.981570 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:18.961547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59313 (local address 127.0.97.62:43339)
0929 02:46:18.961961 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:18.961965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:18.962926 (+ 961us) server_negotiation.cc:407] Connection header received
0929 02:46:18.964752 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:18.964758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:18.964852 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:18.964973 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:18.966546 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:18.967328 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:18.969750 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:18.970485 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:18.970635 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:18.974514 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:18.974522 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:18.974529 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:18.974572 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:18.978191 (+ 3619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:18.978940 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:18.978949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:18.978956 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:18.979099 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:18.979511 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:18.979519 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:18.979525 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:18.979895 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:18.980042 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:18.980777 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 02:46:18.981038 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":206,"threads_started":1}
I20260929 02:46:18.982235 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:18.962108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43339 (local address 127.0.97.1:59313)
0929 02:46:18.962760 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:18.962809 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:18.964433 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:18.965209 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:18.965217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:18.965708 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:18.966372 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:18.966400 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:18.967491 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:18.967497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:18.969553 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:18.969570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:18.970859 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:18.970868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:18.971228 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:18.971981 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:18.972004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:18.974269 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:18.978340 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:18.978348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:18.978363 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:18.978804 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:18.979205 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:18.979208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:18.979210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:18.979388 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:18.980019 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:18.980025 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:18.980349 (+ 324us) client_negotiation.cc:769] Sending connection context
0929 02:46:18.980664 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 02:46:18.981075 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":254,"threads_started":1}
I20260929 02:46:18.983939 3664 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43339
I20260929 02:46:18.984308 3664 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:18.985236 3664 heartbeater.cc:511] Master 127.0.97.62:43339 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:46:18.987597 3475 ts_manager.cc:194] Registered new tserver with Master: 9831663acd02411f86f42f4a0ae2a1ab (127.0.97.1:35863)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:46:18.989701 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:19.012317 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43339
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:19.337677 3670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:19.337985 3670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:19.338129 3670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:19.345515 3670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:19.345631 3670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:19.345813 3670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:19.355813 3670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:19.357208 3670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:19.358565 3670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:19.372196 3678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:19.372287 3676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:19.372306 3675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:19.373366 3670 server_base.cc:1034] running on GCE node
I20260929 02:46:19.374082 3670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:19.375113 3670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:19.376335 3670 hybrid_clock.cc:648] HybridClock initialized: now 1790649979376285 us; error 44 us; skew 500 ppm
I20260929 02:46:19.376703 3670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:19 dist-test-slave-6tjh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:19.379853 3670 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:19.381459 3670 webserver.cc:460] Webserver started at http://127.0.97.2:39479/ using document root <none> and password file <none>
I20260929 02:46:19.382169 3670 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:19.382315 3670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:19.382615 3670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:19.385787 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "a7a7be532e504bc1bdc4467575eae98b"
format_stamp: "Formatted at 2026-09-29 02:46:19 on dist-test-slave-6tjh"
server_key: "a6adc6c10a7cad055d4efded22b68ee6"
server_key_iv: "7c40a3bd60bf583c2ae87b808ffd6293"
server_key_version: "encryptionkey@0"
I20260929 02:46:19.386652 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a7a7be532e504bc1bdc4467575eae98b"
format_stamp: "Formatted at 2026-09-29 02:46:19 on dist-test-slave-6tjh"
server_key: "a6adc6c10a7cad055d4efded22b68ee6"
server_key_iv: "7c40a3bd60bf583c2ae87b808ffd6293"
server_key_version: "encryptionkey@0"
I20260929 02:46:19.392122 3670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:46:19.395965 3685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:19.397495 3670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:19.397675 3670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "a7a7be532e504bc1bdc4467575eae98b"
format_stamp: "Formatted at 2026-09-29 02:46:19 on dist-test-slave-6tjh"
server_key: "a6adc6c10a7cad055d4efded22b68ee6"
server_key_iv: "7c40a3bd60bf583c2ae87b808ffd6293"
server_key_version: "encryptionkey@0"
I20260929 02:46:19.397884 3670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:19.413910 3670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:19.417925 3670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:19.418222 3670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:19.419171 3670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:19.420503 3670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:19.420634 3670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 02:46:19.420789 3670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:19.420864 3670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:19.454591 3670 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39787
I20260929 02:46:19.454648 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39787 every 8 connection(s)
I20260929 02:46:19.455941 3670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:19.460668 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3670
I20260929 02:46:19.460865 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:19.461351 388 external_mini_cluster.cc:1428] Setting key 8c87eceb2056872f7764d7c7089ca4cc
Sep 29 02:46:19 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:19.478463 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:19.458772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:50837 (local address 127.0.97.62:43339)
0929 02:46:19.458977 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:19.458984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:19.460425 (+ 1441us) server_negotiation.cc:407] Connection header received
0929 02:46:19.462706 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:19.462711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:19.462782 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:19.462869 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:19.464702 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.465957 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.468240 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.468815 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.468961 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:19.472487 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:19.472492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:19.472495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:19.472529 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:19.475322 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:19.476078 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:19.476084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:19.476088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:19.476212 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:19.476705 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:19.476711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:19.476715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:19.477000 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:19.477155 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:19.477886 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 02:46:19.478114 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:46:19.479538 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:19.459378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43339 (local address 127.0.97.2:50837)
0929 02:46:19.460228 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:19.460278 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:19.462408 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:19.463189 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:19.463202 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:19.463822 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:19.464406 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:19.464423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.466139 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.466143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:19.468066 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:19.468084 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.469058 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.469064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:19.469316 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:19.470034 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:19.470057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:19.472241 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:19.475468 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:19.475478 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:19.475497 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:19.475954 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:19.476340 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:19.476344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:19.476349 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:19.476588 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:19.477140 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:19.477149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:19.477515 (+ 366us) client_negotiation.cc:769] Sending connection context
0929 02:46:19.477819 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 02:46:19.478369 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":313,"threads_started":1}
I20260929 02:46:19.481137 3798 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 02:46:19.481496 3798 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:19.482282 3798 heartbeater.cc:511] Master 127.0.97.62:43339 requested a full tablet report, sending...
I20260929 02:46:19.483745 3475 ts_manager.cc:194] Registered new tserver with Master: a7a7be532e504bc1bdc4467575eae98b (127.0.97.2:39787)
I20260929 02:46:19.484647 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:50837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:19.506314 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43339
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:19.833889 3803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:19.834172 3803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:19.834332 3803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:19.841756 3803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:19.841887 3803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:19.842072 3803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:19.852059 3803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:19.853480 3803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:19.854842 3803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:19.868760 3808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:19.868862 3811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:19.868819 3809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:19.869835 3803 server_base.cc:1034] running on GCE node
I20260929 02:46:19.870520 3803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:19.871518 3803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:19.872710 3803 hybrid_clock.cc:648] HybridClock initialized: now 1790649979872664 us; error 44 us; skew 500 ppm
I20260929 02:46:19.873072 3803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:19 dist-test-slave-6tjh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:19.876173 3803 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:19.877738 3803 webserver.cc:460] Webserver started at http://127.0.97.3:33261/ using document root <none> and password file <none>
I20260929 02:46:19.878448 3803 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:19.878582 3803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:19.878901 3803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:19.881938 3803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "4f7c0e14ae804765bad2da4f3a5177eb"
format_stamp: "Formatted at 2026-09-29 02:46:19 on dist-test-slave-6tjh"
server_key: "fcc1c42c62609118e1bac8769289f103"
server_key_iv: "0af340c7869da4a9a6e3d52dcd0010da"
server_key_version: "encryptionkey@0"
I20260929 02:46:19.882750 3803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4f7c0e14ae804765bad2da4f3a5177eb"
format_stamp: "Formatted at 2026-09-29 02:46:19 on dist-test-slave-6tjh"
server_key: "fcc1c42c62609118e1bac8769289f103"
server_key_iv: "0af340c7869da4a9a6e3d52dcd0010da"
server_key_version: "encryptionkey@0"
I20260929 02:46:19.888037 3803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260929 02:46:19.891743 3818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:19.893194 3803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:46:19.893381 3803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "4f7c0e14ae804765bad2da4f3a5177eb"
format_stamp: "Formatted at 2026-09-29 02:46:19 on dist-test-slave-6tjh"
server_key: "fcc1c42c62609118e1bac8769289f103"
server_key_iv: "0af340c7869da4a9a6e3d52dcd0010da"
server_key_version: "encryptionkey@0"
I20260929 02:46:19.893556 3803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:19.913161 3803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:19.917325 3803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:19.917644 3803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:19.918660 3803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:19.920025 3803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:19.920104 3803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:19.920241 3803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:19.920301 3803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:19.954512 3803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40737
I20260929 02:46:19.954555 3930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40737 every 8 connection(s)
I20260929 02:46:19.955904 3803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:19.965409 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3803
I20260929 02:46:19.965608 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:19.966207 388 external_mini_cluster.cc:1428] Setting key d6ebee06484abb32cb90e25cb8a3db29
Sep 29 02:46:19 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:19.979457 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:19.959616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36993 (local address 127.0.97.62:43339)
0929 02:46:19.959834 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:19.959840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:19.961161 (+ 1321us) server_negotiation.cc:407] Connection header received
0929 02:46:19.962992 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:19.962996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:19.963066 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:19.963171 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:19.964784 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.966210 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.968130 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.968812 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.968931 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:19.973197 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:19.973202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:19.973206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:19.973240 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:19.976379 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:19.977164 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:19.977171 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:19.977174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:19.977301 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:19.977731 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:19.977736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:19.977739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:19.978016 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:19.978176 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:19.978887 (+ 711us) server_negotiation.cc:299] Negotiation successful
0929 02:46:19.979105 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:46:19.980681 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:19.960124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43339 (local address 127.0.97.3:36993)
0929 02:46:19.960985 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:19.961035 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:19.962712 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:19.963401 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:19.963409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:19.963923 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:19.964502 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:19.964520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.966414 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.966418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:19.967975 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:19.967986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:19.969054 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:19.969062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:19.969353 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:19.970136 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:19.970164 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:19.972964 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:19.976536 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:19.976588 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:19.976608 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:19.977032 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:19.977412 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:19.977417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:19.977421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:19.977609 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:19.978147 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:19.978163 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:19.978509 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 02:46:19.978777 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 02:46:19.979333 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":383,"threads_started":1}
I20260929 02:46:19.982227 3931 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43339
I20260929 02:46:19.982576 3931 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:19.983287 3931 heartbeater.cc:511] Master 127.0.97.62:43339 requested a full tablet report, sending...
I20260929 02:46:19.984645 3475 ts_manager.cc:194] Registered new tserver with Master: 4f7c0e14ae804765bad2da4f3a5177eb (127.0.97.3:40737)
I20260929 02:46:19.985492 3475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36993
I20260929 02:46:19.993805 3664 heartbeater.cc:503] Master 127.0.97.62:43339 was elected leader, sending a full tablet report...
I20260929 02:46:19.995060 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:20.045779 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:20.031324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42072 (local address 127.0.97.62:43339)
0929 02:46:20.031585 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:20.031593 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:20.031890 (+ 297us) server_negotiation.cc:407] Connection header received
0929 02:46:20.032125 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:20.032130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:20.032199 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:20.032314 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:20.033795 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.034529 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.035793 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.036204 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.036350 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:20.040217 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:20.040222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:20.040226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:20.040259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:20.043011 (+ 2752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.043704 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.043711 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.043719 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.043844 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.044240 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.044246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.044249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.044677 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:20.044926 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:20.045227 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:46:20.045453 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260929 02:46:20.046362 3474 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42072
I20260929 02:46:20.048105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3536
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:20.064713 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:20.049095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40532 (local address 127.0.97.1:35863)
0929 02:46:20.049671 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:20.049678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:20.049702 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:46:20.049758 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:20.049764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:20.049992 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:20.050128 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:20.051229 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.052290 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.053633 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.054195 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.054326 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:20.058490 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:20.058502 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:20.058517 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:20.058553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:20.061708 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.062416 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.062430 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.062432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.062572 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.063011 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.063017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.063021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.063377 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:20.063690 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:20.064007 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 02:46:20.064287 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":287,"threads_started":1}
W20260929 02:46:20.066066 3659 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40532
W20260929 02:46:20.067317 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3536: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:20.067524 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3536
I20260929 02:46:20.083292 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3670
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:46:20.099982 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:20.084220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.0.97.2:39787)
0929 02:46:20.084806 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:20.084815 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:20.084869 (+ 54us) server_negotiation.cc:407] Connection header received
0929 02:46:20.084956 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:20.084962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:20.085208 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:20.085345 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:20.086702 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.087743 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.089110 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.089694 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.089908 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:20.094072 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:20.094083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:20.094099 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:20.094136 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:20.097039 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.097760 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.097771 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.097774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.097947 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.098414 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.098418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.098420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.098798 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:20.099048 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:20.099355 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 02:46:20.099553 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":292,"threads_started":1}
W20260929 02:46:20.101495 3793 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54336
W20260929 02:46:20.102911 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:20.103052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3670
I20260929 02:46:20.118590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3803
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649980, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:46:20.136453 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:20.119830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47096 (local address 127.0.97.3:40737)
0929 02:46:20.120526 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:20.120534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:20.120640 (+ 106us) server_negotiation.cc:407] Connection header received
0929 02:46:20.120743 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:20.120750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:20.120976 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:20.121115 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:20.122410 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.123513 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.124925 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:20.125508 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:20.125608 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:20.130045 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:20.130059 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:20.130075 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:20.130139 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:20.133631 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.134388 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.134396 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.134398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.134537 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:20.134925 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:20.134930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:20.134932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:20.135310 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:20.135557 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:20.135803 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 02:46:20.136012 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":394,"threads_started":1}
W20260929 02:46:20.137871 3926 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47096
W20260929 02:46:20.139051 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3803: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:46:20.139213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3803
I20260929 02:46:20.155087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3443
I20260929 02:46:20.156395 3505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:20.260483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3443
2026-09-29T02:46:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3625 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3948](info): set up 2 sockets
Sep 29 02:46:20 dist-test-slave-6tjh krb5kdc[3948](info): commencing operation
krb5kdc: starting...
W20260929 02:46:21.373035 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:21 dist-test-slave-6tjh krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:21.567929 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37287
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37287
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:21.897105 3961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:21.897401 3961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:21.897491 3961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:21.904695 3961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:21.904803 3961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:21.904848 3961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:21.904896 3961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:21.904942 3961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:21.914672 3961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37287
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37287
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:21.915922 3961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:21.917274 3961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:21.928691 3966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:21.928882 3967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:21.928978 3969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:21.930140 3961 server_base.cc:1034] running on GCE node
I20260929 02:46:21.931005 3961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:21.932297 3961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:21.933497 3961 hybrid_clock.cc:648] HybridClock initialized: now 1790649981933439 us; error 43 us; skew 500 ppm
I20260929 02:46:21.933799 3961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:21 dist-test-slave-6tjh krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:21.940972 3961 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:21.942471 3961 webserver.cc:460] Webserver started at http://127.0.97.62:38253/ using document root <none> and password file <none>
I20260929 02:46:21.943130 3961 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:21.943228 3961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:21.943480 3961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:21.946363 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "1bf4c34010a04d30b6b5e2ada948347d"
format_stamp: "Formatted at 2026-09-29 02:46:21 on dist-test-slave-6tjh"
server_key: "b8148848bd1f19c5fb0ce43048200dba"
server_key_iv: "db400119f4191b6cadf82f15025e644e"
server_key_version: "encryptionkey@0"
I20260929 02:46:21.947054 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "1bf4c34010a04d30b6b5e2ada948347d"
format_stamp: "Formatted at 2026-09-29 02:46:21 on dist-test-slave-6tjh"
server_key: "b8148848bd1f19c5fb0ce43048200dba"
server_key_iv: "db400119f4191b6cadf82f15025e644e"
server_key_version: "encryptionkey@0"
I20260929 02:46:21.951881 3961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:46:21.955107 3976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:21.956351 3961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:21.956525 3961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "1bf4c34010a04d30b6b5e2ada948347d"
format_stamp: "Formatted at 2026-09-29 02:46:21 on dist-test-slave-6tjh"
server_key: "b8148848bd1f19c5fb0ce43048200dba"
server_key_iv: "db400119f4191b6cadf82f15025e644e"
server_key_version: "encryptionkey@0"
I20260929 02:46:21.956773 3961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:21.989450 3961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:21.993454 3961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:21.993739 3961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:22.011493 3961 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37287
I20260929 02:46:22.011519 4027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37287 every 8 connection(s)
I20260929 02:46:22.012917 3961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:22.015937 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 3961
I20260929 02:46:22.016163 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:22.016891 4028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:22.016978 388 external_mini_cluster.cc:1428] Setting key 923ea262973533efd126ce1a620a2790
I20260929 02:46:22.031273 4028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d: Bootstrap starting.
Sep 29 02:46:22 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:22.035171 4028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:22.036605 4028 log.cc:824] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:22.040005 4028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d: No bootstrap required, opened a new log
I20260929 02:46:22.041296 4031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:22.019006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.97.62:37287)
0929 02:46:22.019771 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:22.019782 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:22.019823 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:46:22.021670 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:22.021692 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:22.022094 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:22.022430 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:22.024202 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:22.025525 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:22.027230 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:22.027914 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:22.028042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:22.032599 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:22.032610 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:22.032631 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:22.032696 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:22.036706 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:22.037537 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:22.037543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:22.037549 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:22.037709 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:22.038137 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:22.038142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:22.038145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:22.038782 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:22.039044 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:22.039503 (+ 459us) server_negotiation.cc:299] Negotiation successful
0929 02:46:22.039900 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":344,"threads_started":1}
I20260929 02:46:22.048257 4028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf4c34010a04d30b6b5e2ada948347d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37287 } }
I20260929 02:46:22.048760 4028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:22.048851 4028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bf4c34010a04d30b6b5e2ada948347d, State: Initialized, Role: FOLLOWER
I20260929 02:46:22.049404 4028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf4c34010a04d30b6b5e2ada948347d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37287 } }
I20260929 02:46:22.049608 4028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:22.049700 4028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:22.049822 4028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:22.051898 4028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf4c34010a04d30b6b5e2ada948347d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37287 } }
I20260929 02:46:22.052417 4028 leader_election.cc:302] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bf4c34010a04d30b6b5e2ada948347d; no voters:
I20260929 02:46:22.053002 4028 leader_election.cc:288] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:22.053126 4033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:22.053893 4033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [term 1 LEADER]: Becoming Leader. State: Replica: 1bf4c34010a04d30b6b5e2ada948347d, State: Running, Role: LEADER
I20260929 02:46:22.054404 4033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf4c34010a04d30b6b5e2ada948347d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37287 } }
I20260929 02:46:22.055190 4028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:22.056895 4034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bf4c34010a04d30b6b5e2ada948347d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf4c34010a04d30b6b5e2ada948347d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37287 } } }
I20260929 02:46:22.057243 4034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:22.058053 4038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:22.057914 4035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bf4c34010a04d30b6b5e2ada948347d. Latest consensus state: current_term: 1 leader_uuid: "1bf4c34010a04d30b6b5e2ada948347d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bf4c34010a04d30b6b5e2ada948347d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37287 } } }
I20260929 02:46:22.058281 4035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:22.063532 4038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:22.074812 4038 catalog_manager.cc:1347] Generated new cluster ID: 89de0687ac0845deb72dcb30aca949d7
I20260929 02:46:22.074927 4038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:22.099542 4038 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:22.101521 4038 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:22.111115 4038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bf4c34010a04d30b6b5e2ada948347d: Generated new TSK 0
I20260929 02:46:22.112229 4038 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:46:22.160305 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37287
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:22.485435 4054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:22.485708 4054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:22.485895 4054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:22.493080 4054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:22.493187 4054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:22.493387 4054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:22.503598 4054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:22.504956 4054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:22.506227 4054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:22.519496 4062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:22.519465 4059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:22.519531 4060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:22.520458 4054 server_base.cc:1034] running on GCE node
I20260929 02:46:22.521113 4054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:22.522058 4054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:22.523248 4054 hybrid_clock.cc:648] HybridClock initialized: now 1790649982523189 us; error 74 us; skew 500 ppm
I20260929 02:46:22.523569 4054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:22 dist-test-slave-6tjh krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:22.526702 4054 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:46:22.528204 4054 webserver.cc:460] Webserver started at http://127.0.97.1:33459/ using document root <none> and password file <none>
I20260929 02:46:22.528985 4054 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:22.529126 4054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:22.529438 4054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:22.532299 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "7a38297ab9c642178f6dd96aa5ca4842"
format_stamp: "Formatted at 2026-09-29 02:46:22 on dist-test-slave-6tjh"
server_key: "c6b514203ac1c6a7cabf1a6d42b80e4c"
server_key_iv: "65fa23924015e508f7bf2bbd71d5a73c"
server_key_version: "encryptionkey@0"
I20260929 02:46:22.533138 4054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7a38297ab9c642178f6dd96aa5ca4842"
format_stamp: "Formatted at 2026-09-29 02:46:22 on dist-test-slave-6tjh"
server_key: "c6b514203ac1c6a7cabf1a6d42b80e4c"
server_key_iv: "65fa23924015e508f7bf2bbd71d5a73c"
server_key_version: "encryptionkey@0"
I20260929 02:46:22.538357 4054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:22.541817 4069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:22.543130 4054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:22.543306 4054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "7a38297ab9c642178f6dd96aa5ca4842"
format_stamp: "Formatted at 2026-09-29 02:46:22 on dist-test-slave-6tjh"
server_key: "c6b514203ac1c6a7cabf1a6d42b80e4c"
server_key_iv: "65fa23924015e508f7bf2bbd71d5a73c"
server_key_version: "encryptionkey@0"
I20260929 02:46:22.543502 4054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:22.585014 4054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:22.588944 4054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:22.589242 4054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:22.590209 4054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:22.591536 4054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:22.591634 4054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:22.591775 4054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:22.591852 4054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:22.624773 4054 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38909
I20260929 02:46:22.624807 4181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38909 every 8 connection(s)
I20260929 02:46:22.626053 4054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:22.629530 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 4054
I20260929 02:46:22.629756 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:22.630533 388 external_mini_cluster.cc:1428] Setting key ec9f3e0a10ebec8de095304768922466
Sep 29 02:46:22 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:22.647974 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:22.628476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45769 (local address 127.0.97.62:37287)
0929 02:46:22.628943 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:22.628948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:22.629963 (+ 1015us) server_negotiation.cc:407] Connection header received
0929 02:46:22.632226 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:22.632232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:22.632317 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:22.632435 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:22.634488 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:22.635190 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:22.637717 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:22.638113 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:22.638219 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:22.641928 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:22.641933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:22.641936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:22.641967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:22.644606 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:22.645441 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:22.645451 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:22.645453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:22.645575 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:22.646037 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:22.646042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:22.646044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:22.646323 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:22.646485 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:22.647330 (+ 845us) server_negotiation.cc:299] Negotiation successful
0929 02:46:22.647568 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":275,"threads_started":1}
I20260929 02:46:22.649199 4185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:22.628960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37287 (local address 127.0.97.1:45769)
0929 02:46:22.629770 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:22.629829 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:22.631937 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:22.632702 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:22.632713 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:22.633461 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:22.634291 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:22.634318 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:22.635366 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:22.635375 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:22.637567 (+ 2192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:22.637585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:22.638344 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:22.638351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:22.638672 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:22.639522 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:22.639546 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:22.641693 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:22.644780 (+ 3087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:22.644790 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:22.644811 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:22.645287 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:22.645699 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:22.645704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:22.645708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:22.645904 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:22.646462 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:22.646471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:22.646892 (+ 421us) client_negotiation.cc:769] Sending connection context
0929 02:46:22.647211 (+ 319us) client_negotiation.cc:240] Negotiation successful
0929 02:46:22.647801 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:46:22.651108 4182 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37287
I20260929 02:46:22.651418 4182 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:22.652215 4182 heartbeater.cc:511] Master 127.0.97.62:37287 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:46:22.654716 3993 ts_manager.cc:194] Registered new tserver with Master: 7a38297ab9c642178f6dd96aa5ca4842 (127.0.97.1:38909)
I20260929 02:46:22.656478 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:45769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:22.676221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37287
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:23.000488 4188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:23.000890 4188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:23.001063 4188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:23.008406 4188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:23.008591 4188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:23.008788 4188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:23.023236 4188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:23.024835 4188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:23.026448 4188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:23.041513 4194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:23.041522 4196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:23.041764 4193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:23.043442 4188 server_base.cc:1034] running on GCE node
I20260929 02:46:23.044641 4188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:23.046327 4188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:23.047576 4188 hybrid_clock.cc:648] HybridClock initialized: now 1790649983047470 us; error 94 us; skew 500 ppm
I20260929 02:46:23.048024 4188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:23.053377 4188 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:23.055276 4188 webserver.cc:460] Webserver started at http://127.0.97.2:46089/ using document root <none> and password file <none>
I20260929 02:46:23.056294 4188 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:23.056456 4188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:23.056879 4188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:23.061327 4188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "7120f53468e44a0aaea868700bd72849"
format_stamp: "Formatted at 2026-09-29 02:46:23 on dist-test-slave-6tjh"
server_key: "1564a1eb28ad25912005c805a4f08d5b"
server_key_iv: "f96118b0532ea37eaafd558eb220b807"
server_key_version: "encryptionkey@0"
I20260929 02:46:23.062371 4188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "7120f53468e44a0aaea868700bd72849"
format_stamp: "Formatted at 2026-09-29 02:46:23 on dist-test-slave-6tjh"
server_key: "1564a1eb28ad25912005c805a4f08d5b"
server_key_iv: "f96118b0532ea37eaafd558eb220b807"
server_key_version: "encryptionkey@0"
I20260929 02:46:23.069943 4188 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260929 02:46:23.075148 4203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:23.077023 4188 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 02:46:23.077219 4188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "7120f53468e44a0aaea868700bd72849"
format_stamp: "Formatted at 2026-09-29 02:46:23 on dist-test-slave-6tjh"
server_key: "1564a1eb28ad25912005c805a4f08d5b"
server_key_iv: "f96118b0532ea37eaafd558eb220b807"
server_key_version: "encryptionkey@0"
I20260929 02:46:23.077447 4188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:23.094542 4188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:23.098690 4188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:23.098996 4188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:23.099975 4188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:23.101540 4188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:23.101645 4188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:23.101796 4188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:23.101881 4188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:23.135854 4188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38427
I20260929 02:46:23.135893 4315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38427 every 8 connection(s)
I20260929 02:46:23.137277 4188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:23.145041 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 4188
I20260929 02:46:23.145272 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:23.145716 388 external_mini_cluster.cc:1428] Setting key 3f4e8bc102870fbb0a2fe22f8edaa771
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:23.159446 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.139946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58621 (local address 127.0.97.62:37287)
0929 02:46:23.140102 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:23.140107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:23.141369 (+ 1262us) server_negotiation.cc:407] Connection header received
0929 02:46:23.143338 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:23.143343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:23.143439 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:23.143551 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:23.145335 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.146203 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.148135 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.148678 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.148832 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.153129 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:23.153134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:23.153138 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:23.153172 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:23.155961 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.156850 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.156856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.156858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.156997 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.157484 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.157496 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.157500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.157782 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:23.157948 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:23.158772 (+ 824us) server_negotiation.cc:299] Negotiation successful
0929 02:46:23.159030 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 02:46:23.160681 4318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.140413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37287 (local address 127.0.97.2:58621)
0929 02:46:23.141193 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:23.141231 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:23.143032 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:23.143785 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:23.143794 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:23.144295 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:23.145077 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:23.145103 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.146386 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.146413 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:23.147966 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:23.147979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.148943 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.148951 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:23.149208 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.150005 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:23.150040 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:23.152871 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:23.156141 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:23.156152 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:23.156169 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:23.156706 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:23.157132 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:23.157137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:23.157141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:23.157354 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:23.157922 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:23.157932 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:23.158323 (+ 391us) client_negotiation.cc:769] Sending connection context
0929 02:46:23.158656 (+ 333us) client_negotiation.cc:240] Negotiation successful
0929 02:46:23.159249 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":361,"threads_started":1}
I20260929 02:46:23.162335 4316 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37287
I20260929 02:46:23.162679 4316 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:46:23.163453 4316 heartbeater.cc:511] Master 127.0.97.62:37287 requested a full tablet report, sending...
I20260929 02:46:23.165129 3993 ts_manager.cc:194] Registered new tserver with Master: 7120f53468e44a0aaea868700bd72849 (127.0.97.2:38427)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 02:46:23.166175 3993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:23.187856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37287
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:23.522169 4321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:23.522500 4321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:23.522603 4321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:23.530169 4321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:23.530306 4321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:23.530431 4321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:23.541078 4321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37287
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:23.542411 4321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:23.543769 4321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:23.558036 4326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:23.558087 4329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:23.558112 4327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:23.558828 4321 server_base.cc:1034] running on GCE node
I20260929 02:46:23.559486 4321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:23.560495 4321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:23.561765 4321 hybrid_clock.cc:648] HybridClock initialized: now 1790649983561709 us; error 70 us; skew 500 ppm
I20260929 02:46:23.562145 4321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:23.565436 4321 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:23.566910 4321 webserver.cc:460] Webserver started at http://127.0.97.3:43403/ using document root <none> and password file <none>
I20260929 02:46:23.567622 4321 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:23.567785 4321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:23.568099 4321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:23.571171 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "8ae3eadf715e4452b228b01ff27724d9"
format_stamp: "Formatted at 2026-09-29 02:46:23 on dist-test-slave-6tjh"
server_key: "bbb639ea3a042e004215d916fccdcd63"
server_key_iv: "3c69a09b90af619f5cd706630bdfff0a"
server_key_version: "encryptionkey@0"
I20260929 02:46:23.571990 4321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8ae3eadf715e4452b228b01ff27724d9"
format_stamp: "Formatted at 2026-09-29 02:46:23 on dist-test-slave-6tjh"
server_key: "bbb639ea3a042e004215d916fccdcd63"
server_key_iv: "3c69a09b90af619f5cd706630bdfff0a"
server_key_version: "encryptionkey@0"
I20260929 02:46:23.577749 4321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 02:46:23.581554 4336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:23.583154 4321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:23.583431 4321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "8ae3eadf715e4452b228b01ff27724d9"
format_stamp: "Formatted at 2026-09-29 02:46:23 on dist-test-slave-6tjh"
server_key: "bbb639ea3a042e004215d916fccdcd63"
server_key_iv: "3c69a09b90af619f5cd706630bdfff0a"
server_key_version: "encryptionkey@0"
I20260929 02:46:23.583654 4321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:23.639250 4321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:23.643639 4321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:23.643968 4321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:23.644982 4321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:23.646283 4321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:23.646361 4321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:23.646533 4321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:23.646601 4321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:23.660004 4182 heartbeater.cc:503] Master 127.0.97.62:37287 was elected leader, sending a full tablet report...
I20260929 02:46:23.680430 4321 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35103
I20260929 02:46:23.680538 4448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35103 every 8 connection(s)
I20260929 02:46:23.681787 4321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:23.687553 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 4321
I20260929 02:46:23.687778 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:23.688278 388 external_mini_cluster.cc:1428] Setting key 919c13c0102e042a683ff33cd6e7e749
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:23.701478 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.684286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51863 (local address 127.0.97.62:37287)
0929 02:46:23.684728 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:23.684733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:23.685716 (+ 983us) server_negotiation.cc:407] Connection header received
0929 02:46:23.687613 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:23.687618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:23.687709 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:23.687832 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:23.689740 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.690770 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.692435 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.692900 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.692994 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.696216 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:23.696220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:23.696224 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:23.696254 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:23.698878 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.699523 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.699526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.699528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.699639 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.699981 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.699984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.699986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.700220 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:23.700331 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:23.701050 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 02:46:23.701201 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":208,"threads_started":1}
I20260929 02:46:23.702704 4452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.684901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37287 (local address 127.0.97.3:51863)
0929 02:46:23.685560 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:23.685595 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:23.687321 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:23.688055 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:23.688064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:23.688728 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:23.689513 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:23.689561 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.690914 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.690919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:23.692306 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:23.692317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.693084 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.693089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:23.693296 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.693927 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:23.693947 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:23.696028 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:23.699001 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:23.699008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:23.699021 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:23.699429 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:23.699727 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:23.699730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:23.699732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:23.699896 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:23.700330 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:23.700335 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:23.700734 (+ 399us) client_negotiation.cc:769] Sending connection context
0929 02:46:23.700973 (+ 239us) client_negotiation.cc:240] Negotiation successful
0929 02:46:23.701355 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":260,"threads_started":1}
I20260929 02:46:23.703985 4449 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37287
I20260929 02:46:23.704249 4449 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:23.704973 4449 heartbeater.cc:511] Master 127.0.97.62:37287 requested a full tablet report, sending...
I20260929 02:46:23.706355 3992 ts_manager.cc:194] Registered new tserver with Master: 8ae3eadf715e4452b228b01ff27724d9 (127.0.97.3:35103)
I20260929 02:46:23.707201 3992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51863
I20260929 02:46:23.716254 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:23.768683 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.753912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.0.97.1:38909)
0929 02:46:23.754494 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:23.754501 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:23.754528 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:46:23.754597 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:23.754604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:23.754883 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:23.755021 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:23.756414 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.757502 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.758659 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.759198 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.759290 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.763181 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:23.763192 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:23.763206 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:23.763242 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:23.766168 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.766737 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.766744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.766746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.766872 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.767171 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.767174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.767176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.767504 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:23.767717 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:23.768026 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:46:23.768217 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":307,"threads_started":1}
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:23.785671 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.772831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56686 (local address 127.0.97.62:37287)
0929 02:46:23.773038 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:23.773046 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:23.773175 (+ 129us) server_negotiation.cc:407] Connection header received
0929 02:46:23.773305 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:23.773309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:23.773372 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:23.773452 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:23.774564 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.775378 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.776686 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.777100 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.777192 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.780909 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:23.780914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:23.780917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:23.780955 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:23.783356 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.783951 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.783955 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.783957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.784078 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.784387 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.784390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.784392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.784752 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:23.784937 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:23.785175 (+ 238us) server_negotiation.cc:299] Negotiation successful
0929 02:46:23.785335 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 02:46:23.786208 3992 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56686
I20260929 02:46:23.787220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 4054
I20260929 02:46:23.799795 4457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.788167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52746 (local address 127.0.97.1:38909)
0929 02:46:23.788370 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:23.788376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:23.788629 (+ 253us) server_negotiation.cc:407] Connection header received
0929 02:46:23.788835 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:23.788842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:23.788932 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:23.789036 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:23.790186 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.790863 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.792049 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.792490 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.792680 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.794884 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:23.794890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:23.794895 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:23.794936 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:23.797359 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.797954 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.797961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.797965 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.798091 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.798455 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.798462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.798465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.798710 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:23.798963 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:23.799252 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 02:46:23.799470 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:46:23.800523 4177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:23.883077 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 4054
I20260929 02:46:23.899513 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 4188
Sep 29 02:46:23 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:46:23.915648 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:23.900484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.0.97.2:38427)
0929 02:46:23.901056 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:23.901063 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:23.901088 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:46:23.901155 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:23.901162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:23.901373 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:23.901553 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:23.902729 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.903759 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.905150 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:23.905717 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:23.905854 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:23.910259 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:23.910271 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:23.910289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:23.910327 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:23.913275 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.913942 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.913949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.913951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.914067 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:23.914380 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:23.914383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:23.914385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:23.914703 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:23.914928 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:23.915169 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 02:46:23.915366 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":309,"threads_started":1}
I20260929 02:46:23.917047 4311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:24.000723 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 4188
I20260929 02:46:24.016067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 4321
Sep 29 02:46:24 dist-test-slave-6tjh krb5kdc[3948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:46:24.033118 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:24.017107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.0.97.3:35103)
0929 02:46:24.017832 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:24.017842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:24.017971 (+ 129us) server_negotiation.cc:407] Connection header received
0929 02:46:24.018198 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:24.018206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:24.018441 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:24.018612 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:24.019880 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:24.020957 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:24.022252 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:24.022801 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:24.022940 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:24.027406 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:24.027416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:24.027431 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:24.027473 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:24.030438 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:24.031098 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:24.031105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:24.031108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:24.031228 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:24.031579 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:24.031586 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:24.031589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:24.031919 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:24.032250 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:24.032485 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 02:46:24.032774 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":391,"threads_started":1}
I20260929 02:46:24.034340 4444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:24.116436 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 4321
I20260929 02:46:24.132037 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 3961
I20260929 02:46:24.133311 4023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:24.236020 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 3961
2026-09-29T02:46:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3976 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:24 dist-test-slave-6tjh krb5kdc[4469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:24 dist-test-slave-6tjh krb5kdc[4469](info): set up 2 sockets
Sep 29 02:46:24 dist-test-slave-6tjh krb5kdc[4469](info): commencing operation
krb5kdc: starting...
W20260929 02:46:25.325068 388 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:25 dist-test-slave-6tjh krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:25.533928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38339
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:25.856158 4482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:25.856468 4482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:25.856666 4482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:25.863919 4482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:25.864041 4482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:25.864121 4482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:25.864215 4482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:25.864327 4482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:25.874471 4482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:25.875813 4482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:25.877170 4482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:25.888932 4487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:25.888976 4488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:25.888950 4490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:25.889961 4482 server_base.cc:1034] running on GCE node
I20260929 02:46:25.890717 4482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:25.892113 4482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:25.893325 4482 hybrid_clock.cc:648] HybridClock initialized: now 1790649985893270 us; error 43 us; skew 500 ppm
I20260929 02:46:25.893695 4482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:25 dist-test-slave-6tjh krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:25.896955 4482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:25.898118 4482 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:25.898252 4482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:25.898574 4482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:25.901542 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "aa0ec2bebe034fa284012f3050720ff9"
format_stamp: "Formatted at 2026-09-29 02:46:25 on dist-test-slave-6tjh"
server_key: "bc231f9da9bb142cbb663349b41f3610"
server_key_iv: "5f0a411db9a2aef0190c3e63c92ffaae"
server_key_version: "encryptionkey@0"
I20260929 02:46:25.902333 4482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "aa0ec2bebe034fa284012f3050720ff9"
format_stamp: "Formatted at 2026-09-29 02:46:25 on dist-test-slave-6tjh"
server_key: "bc231f9da9bb142cbb663349b41f3610"
server_key_iv: "5f0a411db9a2aef0190c3e63c92ffaae"
server_key_version: "encryptionkey@0"
I20260929 02:46:25.907266 4482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:46:25.910832 4496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:25.912240 4482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:25.912431 4482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "aa0ec2bebe034fa284012f3050720ff9"
format_stamp: "Formatted at 2026-09-29 02:46:25 on dist-test-slave-6tjh"
server_key: "bc231f9da9bb142cbb663349b41f3610"
server_key_iv: "5f0a411db9a2aef0190c3e63c92ffaae"
server_key_version: "encryptionkey@0"
I20260929 02:46:25.912732 4482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:25.926452 4482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:25.930598 4482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:25.930871 4482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:25.949288 4482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38339
I20260929 02:46:25.949330 4547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38339 every 8 connection(s)
I20260929 02:46:25.950253 4482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:25.953197 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 4482
I20260929 02:46:25.953442 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:25.954064 388 external_mini_cluster.cc:1428] Setting key 960935b783913e06914c19639e351c3a
I20260929 02:46:25.954133 4548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:46:25 dist-test-slave-6tjh krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:25.970350 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9: Bootstrap starting.
I20260929 02:46:25.973788 4548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:25.974876 4548 log.cc:824] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:25.977674 4551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:25.955997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.0.97.62:38339)
0929 02:46:25.956787 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:25.956801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:25.956849 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:46:25.959019 (+ 2170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:25.959048 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:25.959588 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:25.960072 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:25.961679 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:25.963587 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:25.964992 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:25.965819 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:25.966024 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:25.970172 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:25.970180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:25.970197 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:25.970235 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:25.973466 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:25.974316 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:25.974324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:25.974331 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:25.974498 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:25.974897 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:25.974901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:25.974904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:25.975503 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:25.975829 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:25.976264 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 02:46:25.976716 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":309,"threads_started":1}
I20260929 02:46:25.977896 4548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9: No bootstrap required, opened a new log
I20260929 02:46:25.985111 4548 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0ec2bebe034fa284012f3050720ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260929 02:46:25.985494 4548 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:25.985603 4548 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0ec2bebe034fa284012f3050720ff9, State: Initialized, Role: FOLLOWER
I20260929 02:46:25.986207 4548 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0ec2bebe034fa284012f3050720ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260929 02:46:25.986441 4548 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:25.986579 4548 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:25.986758 4548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:25.989061 4548 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0ec2bebe034fa284012f3050720ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260929 02:46:25.989579 4548 leader_election.cc:302] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa0ec2bebe034fa284012f3050720ff9; no voters:
I20260929 02:46:25.990136 4548 leader_election.cc:288] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:25.990309 4553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:25.990998 4553 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [term 1 LEADER]: Becoming Leader. State: Replica: aa0ec2bebe034fa284012f3050720ff9, State: Running, Role: LEADER
I20260929 02:46:25.991472 4553 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0ec2bebe034fa284012f3050720ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } }
I20260929 02:46:25.992203 4548 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:25.994351 4555 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0ec2bebe034fa284012f3050720ff9. Latest consensus state: current_term: 1 leader_uuid: "aa0ec2bebe034fa284012f3050720ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0ec2bebe034fa284012f3050720ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } } }
I20260929 02:46:25.994683 4555 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:25.994488 4554 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa0ec2bebe034fa284012f3050720ff9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa0ec2bebe034fa284012f3050720ff9" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38339 } } }
I20260929 02:46:25.994834 4554 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:25.999150 4558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:26.003413 4558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:26.011823 4558 catalog_manager.cc:1347] Generated new cluster ID: c830cb018149473db172b058f2cff7cc
I20260929 02:46:26.011989 4558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:26.043184 4558 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:26.044538 4558 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:26.064280 4558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9: Generated new TSK 0
I20260929 02:46:26.065399 4558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:46:26.125404 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:26.453749 4574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:26.454036 4574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:26.454206 4574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:26.461361 4574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:26.461483 4574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:26.461686 4574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:26.471505 4574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:26.472816 4574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:26.474114 4574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:26.486707 4579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:26.486611 4582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:26.486624 4580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:26.487526 4574 server_base.cc:1034] running on GCE node
I20260929 02:46:26.488106 4574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:26.489122 4574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:26.490355 4574 hybrid_clock.cc:648] HybridClock initialized: now 1790649986490285 us; error 67 us; skew 500 ppm
I20260929 02:46:26.490686 4574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:26 dist-test-slave-6tjh krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:26.494102 4574 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:46:26.495266 4574 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:26.495396 4574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:26.495707 4574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:26.499202 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "09b02767c6c94d50bc9bc7ad115f32bb"
format_stamp: "Formatted at 2026-09-29 02:46:26 on dist-test-slave-6tjh"
server_key: "d3e31ed815b1bbc0eaf010ef11d17490"
server_key_iv: "ad102c1e80e483d04fc35b5101a0b4c1"
server_key_version: "encryptionkey@0"
I20260929 02:46:26.500108 4574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "09b02767c6c94d50bc9bc7ad115f32bb"
format_stamp: "Formatted at 2026-09-29 02:46:26 on dist-test-slave-6tjh"
server_key: "d3e31ed815b1bbc0eaf010ef11d17490"
server_key_iv: "ad102c1e80e483d04fc35b5101a0b4c1"
server_key_version: "encryptionkey@0"
I20260929 02:46:26.505076 4574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:46:26.508366 4588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:26.509707 4574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:26.509884 4574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "09b02767c6c94d50bc9bc7ad115f32bb"
format_stamp: "Formatted at 2026-09-29 02:46:26 on dist-test-slave-6tjh"
server_key: "d3e31ed815b1bbc0eaf010ef11d17490"
server_key_iv: "ad102c1e80e483d04fc35b5101a0b4c1"
server_key_version: "encryptionkey@0"
I20260929 02:46:26.510088 4574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:26.527436 4574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:26.532212 4574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:26.532507 4574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:26.533587 4574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:26.535018 4574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:26.535136 4574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:26.535293 4574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:26.535379 4574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:26.568334 4574 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46801
I20260929 02:46:26.568385 4700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46801 every 8 connection(s)
I20260929 02:46:26.569206 4574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:26.576382 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 4574
I20260929 02:46:26.576625 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:26.577514 388 external_mini_cluster.cc:1428] Setting key f9c934f23f9b91eac0da3ac53bfb5eba
Sep 29 02:46:26 dist-test-slave-6tjh krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:26.591019 4703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:26.571651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48669 (local address 127.0.97.62:38339)
0929 02:46:26.572104 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:26.572110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:26.573232 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 02:46:26.574968 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:26.574973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:26.575041 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:26.575140 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:26.576746 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:26.577909 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:26.579745 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:26.580388 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:26.580539 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:26.584158 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:26.584168 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:26.584176 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:26.584226 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:26.587452 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:26.588513 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:26.588517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:26.588519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:26.588742 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:26.589241 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:26.589244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:26.589247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:26.589525 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:26.589634 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:26.590576 (+ 942us) server_negotiation.cc:299] Negotiation successful
0929 02:46:26.590749 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":214,"threads_started":1}
I20260929 02:46:26.592751 4704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:26.572288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38339 (local address 127.0.97.1:48669)
0929 02:46:26.573063 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:26.573106 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:26.574690 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:26.575342 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:26.575351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:26.575887 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:26.576487 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:26.576519 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:26.578062 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:26.578066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:26.579598 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:26.579611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:26.580674 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:26.580680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:26.580882 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:26.581610 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:26.581634 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:26.583877 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:26.587628 (+ 3751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:26.587640 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:26.587664 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:26.588355 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:26.588852 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:26.588860 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:26.588866 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:26.589116 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:26.589654 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:26.589666 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:26.590175 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 02:46:26.590484 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 02:46:26.591114 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":368,"threads_started":1}
I20260929 02:46:26.594822 4701 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38339
I20260929 02:46:26.595194 4701 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:26.595999 4701 heartbeater.cc:511] Master 127.0.97.62:38339 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:46:26.598642 4513 ts_manager.cc:194] Registered new tserver with Master: 09b02767c6c94d50bc9bc7ad115f32bb (127.0.97.1:46801)
I20260929 02:46:26.600310 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:26.629860 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:26.957324 4707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:26.957736 4707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:26.957918 4707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:26.969988 4707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:26.970144 4707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:26.970371 4707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:26.981437 4707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:26.982761 4707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:26.984062 4707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:26.997748 4712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:26.997808 4715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:26.997723 4713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:26.998618 4707 server_base.cc:1034] running on GCE node
I20260929 02:46:26.999432 4707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:27.001010 4707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:27.002230 4707 hybrid_clock.cc:648] HybridClock initialized: now 1790649987002163 us; error 62 us; skew 500 ppm
I20260929 02:46:27.002568 4707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:27 dist-test-slave-6tjh krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:27.009474 4707 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:27.010648 4707 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:27.010741 4707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:27.011005 4707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:27.014087 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "30b7a4a61bee46ebb50d564c3aae5f8c"
format_stamp: "Formatted at 2026-09-29 02:46:27 on dist-test-slave-6tjh"
server_key: "121349f77e64161c5d97f1fb62a63f99"
server_key_iv: "9b10c0985d5b55da4bbeec4c4549bafd"
server_key_version: "encryptionkey@0"
I20260929 02:46:27.014921 4707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "30b7a4a61bee46ebb50d564c3aae5f8c"
format_stamp: "Formatted at 2026-09-29 02:46:27 on dist-test-slave-6tjh"
server_key: "121349f77e64161c5d97f1fb62a63f99"
server_key_iv: "9b10c0985d5b55da4bbeec4c4549bafd"
server_key_version: "encryptionkey@0"
I20260929 02:46:27.020164 4707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:46:27.023703 4721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:27.025269 4707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:27.025457 4707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "30b7a4a61bee46ebb50d564c3aae5f8c"
format_stamp: "Formatted at 2026-09-29 02:46:27 on dist-test-slave-6tjh"
server_key: "121349f77e64161c5d97f1fb62a63f99"
server_key_iv: "9b10c0985d5b55da4bbeec4c4549bafd"
server_key_version: "encryptionkey@0"
I20260929 02:46:27.025688 4707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:27.051180 4707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:27.055666 4707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:27.055968 4707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:27.056977 4707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:27.058307 4707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:27.058394 4707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:27.058516 4707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:27.058604 4707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:27.090955 4707 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34593
I20260929 02:46:27.091039 4833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34593 every 8 connection(s)
I20260929 02:46:27.091784 4707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:27.099332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 4707
I20260929 02:46:27.099545 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:27.100076 388 external_mini_cluster.cc:1428] Setting key 383963dd544e3c3677bddbd1488c15b3
Sep 29 02:46:27 dist-test-slave-6tjh krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:27.112707 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.094451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36229 (local address 127.0.97.62:38339)
0929 02:46:27.094871 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.094878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.095885 (+ 1007us) server_negotiation.cc:407] Connection header received
0929 02:46:27.097648 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.097652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.097726 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.097823 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:27.099233 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.100293 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.102448 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.103142 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.103303 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.106681 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:27.106689 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:27.106695 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:27.106740 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:27.109451 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.110376 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.110380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.110382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.110494 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.110954 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.110957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.110959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.111209 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:27.111309 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.112214 (+ 905us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.112371 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":193,"threads_started":1}
I20260929 02:46:27.114300 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.095057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38339 (local address 127.0.97.2:36229)
0929 02:46:27.095726 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:27.095769 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:27.097403 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:27.098020 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:27.098028 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:27.098496 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:27.099043 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.099059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.100426 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.100430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.102313 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.102325 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.103361 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.103365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.103556 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.104234 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:27.104258 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:27.106444 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:27.109587 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:27.109599 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:27.109620 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:27.110243 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:27.110595 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:27.110603 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:27.110609 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:27.110845 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:27.111329 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:27.111340 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:27.111803 (+ 463us) client_negotiation.cc:769] Sending connection context
0929 02:46:27.112128 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 02:46:27.112766 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":277,"threads_started":1}
I20260929 02:46:27.115836 4834 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38339
I20260929 02:46:27.116163 4834 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:27.117041 4834 heartbeater.cc:511] Master 127.0.97.62:38339 requested a full tablet report, sending...
I20260929 02:46:27.118682 4513 ts_manager.cc:194] Registered new tserver with Master: 30b7a4a61bee46ebb50d564c3aae5f8c (127.0.97.2:34593)
I20260929 02:46:27.119900 4513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36229
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:27.148239 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:27.469071 4840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:27.469360 4840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:27.469476 4840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:27.476691 4840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:27.476811 4840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:27.476926 4840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:27.486786 4840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:27.488070 4840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:27.489390 4840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:27.502622 4845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:27.502668 4848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:27.502707 4846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:27.503722 4840 server_base.cc:1034] running on GCE node
I20260929 02:46:27.504741 4840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:27.505784 4840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:27.507045 4840 hybrid_clock.cc:648] HybridClock initialized: now 1790649987506969 us; error 61 us; skew 500 ppm
I20260929 02:46:27.507423 4840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:27 dist-test-slave-6tjh krb5kdc[4469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:27.510649 4840 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:27.511816 4840 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:27.511953 4840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:27.512282 4840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:27.515493 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "ea08111ec81a4662911839a9cb1f17e4"
format_stamp: "Formatted at 2026-09-29 02:46:27 on dist-test-slave-6tjh"
server_key: "6ef02c5611ad7006f7403b285c1cb5f8"
server_key_iv: "c27dd527bf7d6862aa9240b6ffc0815d"
server_key_version: "encryptionkey@0"
I20260929 02:46:27.516430 4840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ea08111ec81a4662911839a9cb1f17e4"
format_stamp: "Formatted at 2026-09-29 02:46:27 on dist-test-slave-6tjh"
server_key: "6ef02c5611ad7006f7403b285c1cb5f8"
server_key_iv: "c27dd527bf7d6862aa9240b6ffc0815d"
server_key_version: "encryptionkey@0"
I20260929 02:46:27.521740 4840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:27.525470 4854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:27.527107 4840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:27.527329 4840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "ea08111ec81a4662911839a9cb1f17e4"
format_stamp: "Formatted at 2026-09-29 02:46:27 on dist-test-slave-6tjh"
server_key: "6ef02c5611ad7006f7403b285c1cb5f8"
server_key_iv: "c27dd527bf7d6862aa9240b6ffc0815d"
server_key_version: "encryptionkey@0"
I20260929 02:46:27.527577 4840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:27.541049 4840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:27.545290 4840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:27.545594 4840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:27.546615 4840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:27.548040 4840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:27.548169 4840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:27.548316 4840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:27.548406 4840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:27.588688 4840 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40837
I20260929 02:46:27.588781 4966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40837 every 8 connection(s)
I20260929 02:46:27.589625 4840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:27.597384 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 4840
I20260929 02:46:27.597604 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:27.598042 388 external_mini_cluster.cc:1428] Setting key 44da067c3b875a2cdd6a110276369fd2
Sep 29 02:46:27 dist-test-slave-6tjh krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:27.604713 4701 heartbeater.cc:503] Master 127.0.97.62:38339 was elected leader, sending a full tablet report...
I20260929 02:46:27.610796 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.592152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60695 (local address 127.0.97.62:38339)
0929 02:46:27.592379 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.592385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.593775 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 02:46:27.595607 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.595613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.595731 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.595845 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:27.597735 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.598633 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.600635 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.601199 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.601363 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.604896 (+ 3533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:27.604902 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:27.604907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:27.604949 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:27.607752 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.608488 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.608494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.608498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.608677 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.609102 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.609106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.609110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.609399 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:27.609555 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.610251 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.610474 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:46:27.611939 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.592732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38339 (local address 127.0.97.3:60695)
0929 02:46:27.593610 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:27.593650 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:27.595322 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:27.596086 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:27.596096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:27.596690 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:27.597480 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.597510 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.598848 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.598854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.600402 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.600414 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.601450 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.601457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.601713 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.602375 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:27.602406 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:27.604689 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:27.607895 (+ 3206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:27.607903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:27.607921 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:27.608363 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:27.608790 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:27.608795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:27.608799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:27.608984 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:27.609528 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:27.609536 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:27.609897 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 02:46:27.610161 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 02:46:27.610691 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":328,"threads_started":1}
I20260929 02:46:27.613705 4967 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38339
I20260929 02:46:27.613966 4967 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:27.614650 4967 heartbeater.cc:511] Master 127.0.97.62:38339 requested a full tablet report, sending...
I20260929 02:46:27.616052 4512 ts_manager.cc:194] Registered new tserver with Master: ea08111ec81a4662911839a9cb1f17e4 (127.0.97.3:40837)
I20260929 02:46:27.617024 4512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60695
I20260929 02:46:27.626237 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:46:27.643558 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.631119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49008 (local address 127.0.97.62:38339)
0929 02:46:27.631345 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.631355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.631747 (+ 392us) server_negotiation.cc:407] Connection header received
0929 02:46:27.631938 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.631942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.632007 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.632096 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:27.633814 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.634573 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.635800 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.636212 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.636331 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.638563 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:27.638567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:27.638570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:27.638602 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:27.641058 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.641747 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.641751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.641753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.641884 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.642199 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.642203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.642206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.642508 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:27.642723 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.642982 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.643166 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:46:27.648051 4512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49008:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:27.655066 4512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:27.688241 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.668403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.0.97.2:34593)
0929 02:46:27.669006 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.669020 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.680214 (+ 11194us) server_negotiation.cc:407] Connection header received
0929 02:46:27.680639 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.680647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.680927 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.681141 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:27.681850 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.683336 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.685916 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.687010 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.687162 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.687394 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.687725 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.687936 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":354,"threads_started":1}
I20260929 02:46:27.688297 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.667580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46801 (local address 127.0.0.1:42788)
0929 02:46:27.669605 (+ 2025us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:27.669641 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:27.669813 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:27.680824 (+ 11011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:27.680831 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:27.680860 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:27.681349 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.681362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.682961 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.682968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.685491 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.685506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.687301 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.687308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.687505 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.687541 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:46:27.687634 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 02:46:27.687834 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1770,"mutex_wait_us":3,"thread_start_us":1620,"threads_started":1}
I20260929 02:46:27.689028 4981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.670064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40837 (local address 127.0.0.1:52576)
0929 02:46:27.680593 (+ 10529us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:27.680610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:27.680743 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:27.681568 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:27.681572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:27.681597 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:27.681984 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.681995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.683988 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.683995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.686174 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.686187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.687990 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.687995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.688160 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.688190 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:46:27.688309 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 02:46:27.688624 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10360,"mutex_wait_us":27,"thread_start_us":8941,"threads_started":1}
I20260929 02:46:27.689190 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.670456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52576 (local address 127.0.97.3:40837)
0929 02:46:27.679017 (+ 8561us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.679030 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.680716 (+ 1686us) server_negotiation.cc:407] Connection header received
0929 02:46:27.680902 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.680920 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.681131 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.681335 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:27.682169 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.683857 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.686347 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.687772 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.687905 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.688135 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.688364 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.688857 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8319,"thread_start_us":8112,"threads_started":1}
I20260929 02:46:27.689436 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.667747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42788 (local address 127.0.97.1:46801)
0929 02:46:27.680015 (+ 12268us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.680028 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.680068 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:46:27.680203 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.680216 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.680563 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.681401 (+ 838us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:27.681651 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.682829 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.685708 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.687068 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.688593 (+ 1525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.688830 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.688900 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.689074 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12038,"thread_start_us":11850,"threads_started":1}
I20260929 02:46:27.692705 4983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.667927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34593 (local address 127.0.0.1:47682)
0929 02:46:27.680090 (+ 12163us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:27.680110 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:27.680348 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:27.681122 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:27.681127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:27.681155 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:27.681689 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.681699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.683480 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.683485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.685745 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.685759 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.687209 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.687215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.687428 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.687492 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 02:46:27.688754 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0929 02:46:27.688899 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12038,"mutex_wait_us":90,"thread_start_us":12306,"threads_started":1}
I20260929 02:46:27.692860 4902 tablet_service.cc:1467] Processing CreateTablet for tablet 92069a029d084fb9ae6b481446832351 (DEFAULT_TABLE table=test-table [id=b8c1777f55cc47dc9eda17e2598d5e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:27.692873 4636 tablet_service.cc:1467] Processing CreateTablet for tablet 92069a029d084fb9ae6b481446832351 (DEFAULT_TABLE table=test-table [id=b8c1777f55cc47dc9eda17e2598d5e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:27.694134 4636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92069a029d084fb9ae6b481446832351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:27.694134 4902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92069a029d084fb9ae6b481446832351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:27.700377 4769 tablet_service.cc:1467] Processing CreateTablet for tablet 92069a029d084fb9ae6b481446832351 (DEFAULT_TABLE table=test-table [id=b8c1777f55cc47dc9eda17e2598d5e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:27.701943 4769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92069a029d084fb9ae6b481446832351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:27.705516 4985 tablet_bootstrap.cc:492] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Bootstrap starting.
I20260929 02:46:27.708617 4985 tablet_bootstrap.cc:654] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:27.709256 4986 tablet_bootstrap.cc:492] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Bootstrap starting.
I20260929 02:46:27.711199 4985 log.cc:824] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:27.712623 4986 tablet_bootstrap.cc:654] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:27.713826 4986 log.cc:824] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:27.714479 4985 tablet_bootstrap.cc:492] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: No bootstrap required, opened a new log
I20260929 02:46:27.714740 4985 ts_tablet_manager.cc:1397] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:46:27.716116 4989 tablet_bootstrap.cc:492] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Bootstrap starting.
I20260929 02:46:27.716979 4986 tablet_bootstrap.cc:492] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: No bootstrap required, opened a new log
I20260929 02:46:27.717232 4986 ts_tablet_manager.cc:1397] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 02:46:27.718974 4989 tablet_bootstrap.cc:654] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:27.719960 4989 log.cc:824] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:27.723928 4989 tablet_bootstrap.cc:492] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: No bootstrap required, opened a new log
I20260929 02:46:27.724203 4989 ts_tablet_manager.cc:1397] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260929 02:46:27.725625 4985 raft_consensus.cc:348] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.726099 4985 raft_consensus.cc:374] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:27.726182 4985 raft_consensus.cc:729] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea08111ec81a4662911839a9cb1f17e4, State: Initialized, Role: FOLLOWER
I20260929 02:46:27.726941 4985 consensus_queue.cc:260] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.727816 4986 raft_consensus.cc:348] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.728576 4986 raft_consensus.cc:374] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:27.728695 4986 raft_consensus.cc:729] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b02767c6c94d50bc9bc7ad115f32bb, State: Initialized, Role: FOLLOWER
I20260929 02:46:27.729651 4986 consensus_queue.cc:260] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.734724 4989 raft_consensus.cc:348] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.735184 4989 raft_consensus.cc:374] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:27.735262 4989 raft_consensus.cc:729] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b7a4a61bee46ebb50d564c3aae5f8c, State: Initialized, Role: FOLLOWER
I20260929 02:46:27.735805 4985 ts_tablet_manager.cc:1428] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Time spent starting tablet: real 0.021s user 0.012s sys 0.009s
I20260929 02:46:27.735934 4989 consensus_queue.cc:260] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.736310 4967 heartbeater.cc:503] Master 127.0.97.62:38339 was elected leader, sending a full tablet report...
I20260929 02:46:27.738266 4986 ts_tablet_manager.cc:1428] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260929 02:46:27.743803 4834 heartbeater.cc:503] Master 127.0.97.62:38339 was elected leader, sending a full tablet report...
I20260929 02:46:27.744118 4989 ts_tablet_manager.cc:1428] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260929 02:46:27.792975 4993 raft_consensus.cc:482] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:27.793282 4993 raft_consensus.cc:504] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.795096 4993 leader_election.cc:288] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09b02767c6c94d50bc9bc7ad115f32bb (127.0.97.1:46801), ea08111ec81a4662911839a9cb1f17e4 (127.0.97.3:40837)
I20260929 02:46:27.802729 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.795694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45521 (local address 127.0.97.1:46801)
0929 02:46:27.795924 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.795929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.796524 (+ 595us) server_negotiation.cc:407] Connection header received
0929 02:46:27.796743 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.796752 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.796836 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.796946 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:27.797679 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.798704 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.800731 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.801690 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.801801 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.801879 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.802324 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.802475 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:46:27.802842 4994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.795525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46801 (local address 127.0.97.2:45521)
0929 02:46:27.796383 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:27.796406 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:27.796581 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:27.797062 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:27.797070 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:27.797105 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:27.797512 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.797526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.798845 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.798851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.800542 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.800578 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.801924 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.801931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.802115 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.802154 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:46:27.802265 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:46:27.802420 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"spinlock_wait_cycles":30976,"thread_start_us":500,"threads_started":1}
I20260929 02:46:27.803254 4995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.796519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40837 (local address 127.0.97.2:54483)
0929 02:46:27.797141 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:27.797161 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:27.797280 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:27.797619 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:27.797624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:27.797687 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:27.798080 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.798092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.799342 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.799347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.801042 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:27.801058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.802451 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.802457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:27.802643 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.802673 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:46:27.802778 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 02:46:27.802931 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":326,"threads_started":1}
I20260929 02:46:27.803426 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.796721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54483 (local address 127.0.97.3:40837)
0929 02:46:27.796969 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.796975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.797262 (+ 287us) server_negotiation.cc:407] Connection header received
0929 02:46:27.797368 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.797375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.797456 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.797562 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:27.798265 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.799192 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.801213 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.802242 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.802356 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.802451 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.803008 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.803173 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 02:46:27.803978 4656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92069a029d084fb9ae6b481446832351" candidate_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" is_pre_election: true
I20260929 02:46:27.804281 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92069a029d084fb9ae6b481446832351" candidate_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea08111ec81a4662911839a9cb1f17e4" is_pre_election: true
I20260929 02:46:27.804482 4656 raft_consensus.cc:2454] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b7a4a61bee46ebb50d564c3aae5f8c in term 0.
I20260929 02:46:27.804821 4922 raft_consensus.cc:2454] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 30b7a4a61bee46ebb50d564c3aae5f8c in term 0.
I20260929 02:46:27.805188 4723 leader_election.cc:302] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b02767c6c94d50bc9bc7ad115f32bb, 30b7a4a61bee46ebb50d564c3aae5f8c; no voters:
I20260929 02:46:27.805576 4993 raft_consensus.cc:2781] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:27.805729 4993 raft_consensus.cc:482] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:27.805825 4993 raft_consensus.cc:3037] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:27.807996 4993 raft_consensus.cc:504] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.808662 4993 leader_election.cc:288] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [CANDIDATE]: Term 1 election: Requested vote from peers 09b02767c6c94d50bc9bc7ad115f32bb (127.0.97.1:46801), ea08111ec81a4662911839a9cb1f17e4 (127.0.97.3:40837)
I20260929 02:46:27.809147 4656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92069a029d084fb9ae6b481446832351" candidate_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09b02767c6c94d50bc9bc7ad115f32bb"
I20260929 02:46:27.809212 4922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92069a029d084fb9ae6b481446832351" candidate_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea08111ec81a4662911839a9cb1f17e4"
I20260929 02:46:27.809374 4656 raft_consensus.cc:3037] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:27.809482 4922 raft_consensus.cc:3037] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:27.811565 4656 raft_consensus.cc:2454] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b7a4a61bee46ebb50d564c3aae5f8c in term 1.
I20260929 02:46:27.811651 4922 raft_consensus.cc:2454] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 30b7a4a61bee46ebb50d564c3aae5f8c in term 1.
I20260929 02:46:27.812038 4723 leader_election.cc:302] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09b02767c6c94d50bc9bc7ad115f32bb, 30b7a4a61bee46ebb50d564c3aae5f8c; no voters:
I20260929 02:46:27.812347 4993 raft_consensus.cc:2781] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:27.812893 4993 raft_consensus.cc:686] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 1 LEADER]: Becoming Leader. State: Replica: 30b7a4a61bee46ebb50d564c3aae5f8c, State: Running, Role: LEADER
I20260929 02:46:27.813452 4993 consensus_queue.cc:237] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } }
I20260929 02:46:27.818199 4512 catalog_manager.cc:5495] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 30b7a4a61bee46ebb50d564c3aae5f8c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30b7a4a61bee46ebb50d564c3aae5f8c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34593 } health_report { overall_health: HEALTHY } } }
W20260929 02:46:27.821487 4702 tablet.cc:2369] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:27.840986 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.835656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47692 (local address 127.0.97.2:34593)
0929 02:46:27.835878 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.835884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.835933 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:46:27.836100 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.836104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.836201 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.836333 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:27.836687 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.837363 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.838666 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.839112 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.839241 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.839672 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:27.840239 (+ 567us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:27.840360 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.840471 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.840607 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 02:46:27.841084 4968 tablet.cc:2369] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:46:27.843626 4835 tablet.cc:2369] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:27.849310 4656 raft_consensus.cc:1261] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 1 FOLLOWER]: Refusing update from remote peer 30b7a4a61bee46ebb50d564c3aae5f8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:27.849395 4922 raft_consensus.cc:1261] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 1 FOLLOWER]: Refusing update from remote peer 30b7a4a61bee46ebb50d564c3aae5f8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:27.850389 4996 consensus_queue.cc:1035] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09b02767c6c94d50bc9bc7ad115f32bb" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:27.850792 4993 consensus_queue.cc:1035] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea08111ec81a4662911839a9cb1f17e4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:27.877107 4512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:27.877394 4512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49008:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:27.881044 4512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa0ec2bebe034fa284012f3050720ff9: Sending DeleteTablet for 3 replicas of tablet 92069a029d084fb9ae6b481446832351
I20260929 02:46:27.882226 4636 tablet_service.cc:1514] Processing DeleteTablet for tablet 92069a029d084fb9ae6b481446832351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42788
I20260929 02:46:27.882254 4902 tablet_service.cc:1514] Processing DeleteTablet for tablet 92069a029d084fb9ae6b481446832351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52576
I20260929 02:46:27.882534 4769 tablet_service.cc:1514] Processing DeleteTablet for tablet 92069a029d084fb9ae6b481446832351 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:27 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47682
I20260929 02:46:27.883401 5011 tablet_replica.cc:330] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: stopping tablet replica
I20260929 02:46:27.883929 5011 raft_consensus.cc:2229] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:27.884140 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 4574
I20260929 02:46:27.884811 5011 raft_consensus.cc:2258] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:27.890529 5011 ts_tablet_manager.cc:1905] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:27.895264 5010 tablet_replica.cc:330] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: stopping tablet replica
I20260929 02:46:27.895730 5010 raft_consensus.cc:2229] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:27.896237 5010 raft_consensus.cc:2258] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:27.897315 5009 tablet_replica.cc:330] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: stopping tablet replica
I20260929 02:46:27.897716 5009 raft_consensus.cc:2229] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:27.898190 5009 raft_consensus.cc:2258] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:27.899955 5010 ts_tablet_manager.cc:1905] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:27.902678 5011 ts_tablet_manager.cc:1918] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:27.902911 5011 log.cc:1196] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/92069a029d084fb9ae6b481446832351
I20260929 02:46:27.903594 5011 ts_tablet_manager.cc:1939] T 92069a029d084fb9ae6b481446832351 P 30b7a4a61bee46ebb50d564c3aae5f8c: Deleting consensus metadata
I20260929 02:46:27.905977 4497 catalog_manager.cc:4841] TS 30b7a4a61bee46ebb50d564c3aae5f8c (127.0.97.2:34593): tablet 92069a029d084fb9ae6b481446832351 (table test-table [id=b8c1777f55cc47dc9eda17e2598d5e11]) successfully deleted
I20260929 02:46:27.907974 5009 ts_tablet_manager.cc:1905] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:27.908064 5010 ts_tablet_manager.cc:1918] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:27.908241 5010 log.cc:1196] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/92069a029d084fb9ae6b481446832351
Sep 29 02:46:27 dist-test-slave-6tjh krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:27.908820 5010 ts_tablet_manager.cc:1939] T 92069a029d084fb9ae6b481446832351 P 09b02767c6c94d50bc9bc7ad115f32bb: Deleting consensus metadata
I20260929 02:46:27.910696 4498 catalog_manager.cc:4841] TS 09b02767c6c94d50bc9bc7ad115f32bb (127.0.97.1:46801): tablet 92069a029d084fb9ae6b481446832351 (table test-table [id=b8c1777f55cc47dc9eda17e2598d5e11]) successfully deleted
I20260929 02:46:27.915805 5009 ts_tablet_manager.cc:1918] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:27.915990 5009 log.cc:1196] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/92069a029d084fb9ae6b481446832351
I20260929 02:46:27.916534 5009 ts_tablet_manager.cc:1939] T 92069a029d084fb9ae6b481446832351 P ea08111ec81a4662911839a9cb1f17e4: Deleting consensus metadata
I20260929 02:46:27.917333 4980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:27.886042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.0.97.1:46801)
0929 02:46:27.886226 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:27.886232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:27.898036 (+ 11804us) server_negotiation.cc:407] Connection header received
0929 02:46:27.898306 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:27.898314 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:27.898396 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:27.898472 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:27.900427 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.901574 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.903467 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:27.904141 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:27.904242 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:27.909803 (+ 5561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:27.909819 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:27.909838 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:27.909882 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:27.913969 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.914983 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.914994 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.914999 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.915165 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:27.915589 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:27.915594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:27.915597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:27.916063 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:27.916482 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:27.916714 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 02:46:27.916939 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:46:27.918007 4696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:27.918337 4498 catalog_manager.cc:4841] TS ea08111ec81a4662911839a9cb1f17e4 (127.0.97.3:40837): tablet 92069a029d084fb9ae6b481446832351 (table test-table [id=b8c1777f55cc47dc9eda17e2598d5e11]) successfully deleted
I20260929 02:46:28.024637 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 4574
I20260929 02:46:28.042311 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 4707
Sep 29 02:46:28 dist-test-slave-6tjh krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:46:28.058817 4982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:28.043101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.0.97.2:34593)
0929 02:46:28.043301 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:28.043307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:28.043331 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:46:28.043512 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:28.043517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:28.043588 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:28.043712 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:28.044902 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:28.045718 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:28.046934 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:28.047358 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:28.047492 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:28.051691 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:28.051709 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:28.051730 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:28.051787 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:28.055681 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:28.056361 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:28.056370 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:28.056374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:28.056699 (+ 325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:28.057166 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:28.057173 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:28.057181 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:28.057636 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:28.057977 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:28.058150 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 02:46:28.058384 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:46:28.059571 4829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:28.177899 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 4707
I20260929 02:46:28.195768 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 4840
Sep 29 02:46:28 dist-test-slave-6tjh krb5kdc[4469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:46:28.210680 4984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:28.196453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52592 (local address 127.0.97.3:40837)
0929 02:46:28.196670 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:28.196676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:28.196753 (+ 77us) server_negotiation.cc:407] Connection header received
0929 02:46:28.196987 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:28.196991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:28.197052 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:28.197163 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:28.198359 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:28.199066 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:28.200349 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:28.200945 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:28.201073 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:28.205270 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:28.205290 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:28.205306 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:28.205339 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:28.207907 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:28.208621 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:28.208632 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:28.208635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:28.208771 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:28.209146 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:28.209152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:28.209156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:28.209523 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:28.209818 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:28.210125 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 02:46:28.210361 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:46:28.211323 4962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:28.299240 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 4840
I20260929 02:46:28.315362 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 4482
I20260929 02:46:28.316593 4543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:28.423298 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 4482
2026-09-29T02:46:28Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4186 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T02:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:28Z Disabled control of system clock
I20260929 02:46:28.489975 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36685
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36685
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:46:28.817142 5020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:28.817451 5020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:28.817592 5020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:28.825170 5020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:28.825299 5020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:28.825380 5020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:28.825472 5020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:28.825533 5020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:28.834992 5020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36685
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:28.836349 5020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:28.838219 5020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:28.850780 5028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:28.850724 5026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:28.850746 5025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:28.851749 5020 server_base.cc:1034] running on GCE node
I20260929 02:46:28.852726 5020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:28.854183 5020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:28.855418 5020 hybrid_clock.cc:648] HybridClock initialized: now 1790649988855373 us; error 45 us; skew 500 ppm
I20260929 02:46:28.855791 5020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:28.858212 5020 webserver.cc:460] Webserver started at http://127.0.97.62:43623/ using document root <none> and password file <none>
I20260929 02:46:28.858953 5020 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:28.859084 5020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:28.859411 5020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:28.862421 5020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "11944bdbc82247be9f0d748d8e2f25ae"
format_stamp: "Formatted at 2026-09-29 02:46:28 on dist-test-slave-6tjh"
server_key: "49b307e3060998d54f5932aa4bcbbab5"
server_key_iv: "d82cd6e3ada56888c6b9d87befccbe83"
server_key_version: "encryptionkey@0"
I20260929 02:46:28.863265 5020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "11944bdbc82247be9f0d748d8e2f25ae"
format_stamp: "Formatted at 2026-09-29 02:46:28 on dist-test-slave-6tjh"
server_key: "49b307e3060998d54f5932aa4bcbbab5"
server_key_iv: "d82cd6e3ada56888c6b9d87befccbe83"
server_key_version: "encryptionkey@0"
I20260929 02:46:28.868611 5020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:46:28.872243 5034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:28.873865 5020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:46:28.874071 5020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "11944bdbc82247be9f0d748d8e2f25ae"
format_stamp: "Formatted at 2026-09-29 02:46:28 on dist-test-slave-6tjh"
server_key: "49b307e3060998d54f5932aa4bcbbab5"
server_key_iv: "d82cd6e3ada56888c6b9d87befccbe83"
server_key_version: "encryptionkey@0"
I20260929 02:46:28.874338 5020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:28.880424 5020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:28.881320 5020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:28.881631 5020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:28.899529 5020 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36685
I20260929 02:46:28.899549 5085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36685 every 8 connection(s)
I20260929 02:46:28.900876 5020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:28.904664 5086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:28.909770 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5020
I20260929 02:46:28.909983 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:28.910594 388 external_mini_cluster.cc:1428] Setting key 63992dc92c23b2ff6573188061e1909f
I20260929 02:46:28.916267 5086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae: Bootstrap starting.
I20260929 02:46:28.920248 5086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:28.920282 5089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:28.912702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.0.97.62:36685)
0929 02:46:28.913899 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:28.913917 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:28.913969 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:46:28.916242 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:28.916277 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:28.916840 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:28.917271 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:28.918331 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:28.918340 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:28.918358 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:28.918376 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:28.918454 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:28.918476 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:28.918697 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:28.918831 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:46:28.919089 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":409,"threads_started":1}
I20260929 02:46:28.921419 5086 log.cc:824] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:28.924155 5086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae: No bootstrap required, opened a new log
I20260929 02:46:28.931586 5086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11944bdbc82247be9f0d748d8e2f25ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260929 02:46:28.931979 5086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:28.932116 5086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11944bdbc82247be9f0d748d8e2f25ae, State: Initialized, Role: FOLLOWER
I20260929 02:46:28.932902 5086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11944bdbc82247be9f0d748d8e2f25ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260929 02:46:28.933166 5086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:28.933308 5086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:28.933496 5086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:28.935711 5086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11944bdbc82247be9f0d748d8e2f25ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260929 02:46:28.936268 5086 leader_election.cc:302] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11944bdbc82247be9f0d748d8e2f25ae; no voters:
I20260929 02:46:28.937120 5086 leader_election.cc:288] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:28.937322 5091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:28.938163 5091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [term 1 LEADER]: Becoming Leader. State: Replica: 11944bdbc82247be9f0d748d8e2f25ae, State: Running, Role: LEADER
I20260929 02:46:28.938759 5091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11944bdbc82247be9f0d748d8e2f25ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } }
I20260929 02:46:28.939389 5086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:28.941907 5093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11944bdbc82247be9f0d748d8e2f25ae. Latest consensus state: current_term: 1 leader_uuid: "11944bdbc82247be9f0d748d8e2f25ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11944bdbc82247be9f0d748d8e2f25ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } } }
I20260929 02:46:28.942402 5093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:28.941932 5092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11944bdbc82247be9f0d748d8e2f25ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11944bdbc82247be9f0d748d8e2f25ae" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36685 } } }
I20260929 02:46:28.942610 5092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:28.944818 5098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:28.950747 5098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:28.960529 5098 catalog_manager.cc:1347] Generated new cluster ID: 9201e18f03e9465a97793abb9b582803
I20260929 02:46:28.960690 5098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:28.977373 5098 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:28.977854 5098 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:28.984756 5098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae: Generated new TSK 0
I20260929 02:46:28.985909 5098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:46:28.998690 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36685
--builtin_ntp_servers=127.0.97.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 02:46:29.333267 5110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:29.333581 5110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:29.333671 5110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:29.341897 5110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:29.342023 5110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:29.342133 5110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:29.351878 5110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:29.353385 5110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:29.355170 5110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:29.370739 5115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:29.370707 5118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:29.370832 5116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:29.371606 5110 server_base.cc:1034] running on GCE node
I20260929 02:46:29.372277 5110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:29.373399 5110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:29.374732 5110 hybrid_clock.cc:648] HybridClock initialized: now 1790649989374697 us; error 32 us; skew 500 ppm
I20260929 02:46:29.375080 5110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:29.377554 5110 webserver.cc:460] Webserver started at http://127.0.97.1:42185/ using document root <none> and password file <none>
I20260929 02:46:29.378423 5110 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:29.378551 5110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:29.378868 5110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:29.381873 5110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "a4a3e396b154486f98316e3ebad16b99"
format_stamp: "Formatted at 2026-09-29 02:46:29 on dist-test-slave-6tjh"
server_key: "a995b75d81932ef8d76bd4dae833ce19"
server_key_iv: "a000d2bb01fdee1f4275c66b83f0e794"
server_key_version: "encryptionkey@0"
I20260929 02:46:29.382705 5110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "a4a3e396b154486f98316e3ebad16b99"
format_stamp: "Formatted at 2026-09-29 02:46:29 on dist-test-slave-6tjh"
server_key: "a995b75d81932ef8d76bd4dae833ce19"
server_key_iv: "a000d2bb01fdee1f4275c66b83f0e794"
server_key_version: "encryptionkey@0"
I20260929 02:46:29.388108 5110 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:46:29.391831 5124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:29.393356 5110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:29.393573 5110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "a4a3e396b154486f98316e3ebad16b99"
format_stamp: "Formatted at 2026-09-29 02:46:29 on dist-test-slave-6tjh"
server_key: "a995b75d81932ef8d76bd4dae833ce19"
server_key_iv: "a000d2bb01fdee1f4275c66b83f0e794"
server_key_version: "encryptionkey@0"
I20260929 02:46:29.393846 5110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:29.399664 5110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:29.400519 5110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:29.400864 5110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:29.401937 5110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:29.403354 5110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:29.403486 5110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:29.403613 5110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:29.403702 5110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:29.436425 5110 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43519
I20260929 02:46:29.436473 5236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43519 every 8 connection(s)
I20260929 02:46:29.437841 5110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:29.439706 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5110
I20260929 02:46:29.439929 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:29.440726 388 external_mini_cluster.cc:1428] Setting key 83bf9d77abb904d2fd41fef0c219e433
I20260929 02:46:29.443742 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36685
--builtin_ntp_servers=127.0.97.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:46:29.447659 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:29.441126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44831 (local address 127.0.97.62:36685)
0929 02:46:29.441937 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:29.441944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:29.443182 (+ 1238us) server_negotiation.cc:407] Connection header received
0929 02:46:29.445308 (+ 2126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:29.445313 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:29.445378 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:29.445482 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:29.446997 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:29.447001 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:29.447004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:29.447021 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:29.447062 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:29.447074 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:29.447171 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:29.447404 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 02:46:29.447456 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":504,"threads_started":1}
I20260929 02:46:29.449251 5240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:29.441862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36685 (local address 127.0.97.1:44831)
0929 02:46:29.443032 (+ 1170us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:29.443074 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:29.445026 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:29.445723 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:29.445732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:29.445836 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:29.446623 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:29.446628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:29.446681 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:29.446804 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:29.447171 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:29.447184 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:29.447195 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:46:29.447488 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 02:46:29.447779 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":336,"threads_started":1}
I20260929 02:46:29.450912 5237 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36685
I20260929 02:46:29.451238 5237 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:29.451946 5237 heartbeater.cc:511] Master 127.0.97.62:36685 requested a full tablet report, sending...
I20260929 02:46:29.454403 5051 ts_manager.cc:194] Registered new tserver with Master: a4a3e396b154486f98316e3ebad16b99 (127.0.97.1:43519)
W20260929 02:46:29.770418 5241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:29.770716 5241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:29.770879 5241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:29.778220 5241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:29.778376 5241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:29.778561 5241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:29.788275 5241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:29.789652 5241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:29.791417 5241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:29.805557 5246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:29.805519 5249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:29.805775 5241 server_base.cc:1034] running on GCE node
W20260929 02:46:29.805593 5247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:29.806658 5241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:29.807570 5241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:29.808804 5241 hybrid_clock.cc:648] HybridClock initialized: now 1790649989808728 us; error 49 us; skew 500 ppm
I20260929 02:46:29.809104 5241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:29.811292 5241 webserver.cc:460] Webserver started at http://127.0.97.2:38745/ using document root <none> and password file <none>
I20260929 02:46:29.811980 5241 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:29.812104 5241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:29.812405 5241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:29.815265 5241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "a80310dc93a84ffd93e24b3e1c20634b"
format_stamp: "Formatted at 2026-09-29 02:46:29 on dist-test-slave-6tjh"
server_key: "5a9adf35cd2afb8a60bcddf9458efe88"
server_key_iv: "6062edb9e9d791c7a2ddd0509ab32214"
server_key_version: "encryptionkey@0"
I20260929 02:46:29.816036 5241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a80310dc93a84ffd93e24b3e1c20634b"
format_stamp: "Formatted at 2026-09-29 02:46:29 on dist-test-slave-6tjh"
server_key: "5a9adf35cd2afb8a60bcddf9458efe88"
server_key_iv: "6062edb9e9d791c7a2ddd0509ab32214"
server_key_version: "encryptionkey@0"
I20260929 02:46:29.821048 5241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:46:29.824368 5255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:29.825701 5241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:29.825886 5241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "a80310dc93a84ffd93e24b3e1c20634b"
format_stamp: "Formatted at 2026-09-29 02:46:29 on dist-test-slave-6tjh"
server_key: "5a9adf35cd2afb8a60bcddf9458efe88"
server_key_iv: "6062edb9e9d791c7a2ddd0509ab32214"
server_key_version: "encryptionkey@0"
I20260929 02:46:29.826115 5241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:29.831555 5241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:29.832321 5241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:29.832636 5241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:29.833491 5241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:29.834854 5241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:29.834970 5241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:29.835098 5241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:29.835183 5241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:29.868306 5241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39781
I20260929 02:46:29.868367 5367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39781 every 8 connection(s)
I20260929 02:46:29.869719 5241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:29.876164 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5241
I20260929 02:46:29.876349 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:29.876959 388 external_mini_cluster.cc:1428] Setting key 70b0f51fe700d1a04a96f7d36fa4d4a2
I20260929 02:46:29.878368 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:29.872377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40193 (local address 127.0.97.62:36685)
0929 02:46:29.872589 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:29.872595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:29.874149 (+ 1554us) server_negotiation.cc:407] Connection header received
0929 02:46:29.875852 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:29.875857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:29.875911 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:29.876002 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:29.877642 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:29.877646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:29.877650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:29.877665 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:29.877697 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:29.877729 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:29.877871 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:29.878089 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 02:46:29.878137 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:46:29.879549 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:29.873057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36685 (local address 127.0.97.2:40193)
0929 02:46:29.873978 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:29.874019 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:29.875528 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:29.876312 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:29.876327 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:29.876462 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:29.877277 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:29.877282 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:29.877334 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:29.877447 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:29.877852 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:29.877863 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:29.877871 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:46:29.878170 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 02:46:29.878434 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":292,"threads_started":1}
I20260929 02:46:29.879829 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36685
--builtin_ntp_servers=127.0.97.20:44013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 02:46:29.881021 5368 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36685
I20260929 02:46:29.881367 5368 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:29.882087 5368 heartbeater.cc:511] Master 127.0.97.62:36685 requested a full tablet report, sending...
I20260929 02:46:29.883658 5051 ts_manager.cc:194] Registered new tserver with Master: a80310dc93a84ffd93e24b3e1c20634b (127.0.97.2:39781)
W20260929 02:46:30.211685 5371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:30.212029 5371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:30.212190 5371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:30.219614 5371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:30.219753 5371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:30.219937 5371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:30.229351 5371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36685
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:30.230701 5371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:30.232565 5371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:30.245852 5376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:30.245782 5377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:30.246078 5379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:30.247293 5371 server_base.cc:1034] running on GCE node
I20260929 02:46:30.247999 5371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:30.249090 5371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:30.250337 5371 hybrid_clock.cc:648] HybridClock initialized: now 1790649990250271 us; error 71 us; skew 500 ppm
I20260929 02:46:30.250684 5371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:30.253180 5371 webserver.cc:460] Webserver started at http://127.0.97.3:44755/ using document root <none> and password file <none>
I20260929 02:46:30.253906 5371 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:30.254035 5371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:30.254469 5371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:30.257472 5371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "1d3263bd37704a61a593d4fa52bf7495"
format_stamp: "Formatted at 2026-09-29 02:46:30 on dist-test-slave-6tjh"
server_key: "cbdb43187e08eeb8042f2aee1f5c7e66"
server_key_iv: "ec41a21ae7a647095770ab32a4458f20"
server_key_version: "encryptionkey@0"
I20260929 02:46:30.258278 5371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1d3263bd37704a61a593d4fa52bf7495"
format_stamp: "Formatted at 2026-09-29 02:46:30 on dist-test-slave-6tjh"
server_key: "cbdb43187e08eeb8042f2aee1f5c7e66"
server_key_iv: "ec41a21ae7a647095770ab32a4458f20"
server_key_version: "encryptionkey@0"
I20260929 02:46:30.263409 5371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 02:46:30.266919 5385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:30.268314 5371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:46:30.268494 5371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "1d3263bd37704a61a593d4fa52bf7495"
format_stamp: "Formatted at 2026-09-29 02:46:30 on dist-test-slave-6tjh"
server_key: "cbdb43187e08eeb8042f2aee1f5c7e66"
server_key_iv: "ec41a21ae7a647095770ab32a4458f20"
server_key_version: "encryptionkey@0"
I20260929 02:46:30.268867 5371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:30.274601 5371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:30.275381 5371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:30.275651 5371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:30.276512 5371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:30.278064 5371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:30.278182 5371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:30.278311 5371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:30.278398 5371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:30.312055 5371 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39841
I20260929 02:46:30.312104 5497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39841 every 8 connection(s)
I20260929 02:46:30.313565 5371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:30.320294 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5371
I20260929 02:46:30.320485 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:30.321110 388 external_mini_cluster.cc:1428] Setting key e1f169325422c4922e0500c43576544c
I20260929 02:46:30.322118 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.316351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35945 (local address 127.0.97.62:36685)
0929 02:46:30.316589 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.316598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.318063 (+ 1465us) server_negotiation.cc:407] Connection header received
0929 02:46:30.319720 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.319725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.319805 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.319917 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.321365 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.321371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.321375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.321393 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.321436 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.321448 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.321567 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.321781 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.321844 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:46:30.322996 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.317019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36685 (local address 127.0.97.3:35945)
0929 02:46:30.317912 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.317946 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:30.319464 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:30.320140 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:30.320147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:30.320235 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:30.321047 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:30.321050 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:30.321084 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:30.321190 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:30.321541 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:30.321550 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:30.321557 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:46:30.321833 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 02:46:30.322069 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":282,"threads_started":1}
I20260929 02:46:30.324383 5498 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36685
I20260929 02:46:30.324690 5498 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:30.325273 5498 heartbeater.cc:511] Master 127.0.97.62:36685 requested a full tablet report, sending...
I20260929 02:46:30.326612 5051 ts_manager.cc:194] Registered new tserver with Master: 1d3263bd37704a61a593d4fa52bf7495 (127.0.97.3:39841)
I20260929 02:46:30.337121 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:46:30.344059 5239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.341720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.0.97.62:36685)
0929 02:46:30.341906 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.341911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.342363 (+ 452us) server_negotiation.cc:407] Connection header received
0929 02:46:30.342491 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.342495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.342545 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.342630 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.343584 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.343589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.343592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.343606 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.343635 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.343646 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.343743 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.343819 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.343862 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:46:30.348441 5051 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:30.355333 5051 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:30.381596 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.367261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35350)
0929 02:46:30.377701 (+ 10440us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.377758 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:30.377985 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:30.378855 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:30.378870 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:30.378978 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:30.380125 (+ 1147us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:30.380131 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:30.380170 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:30.380348 (+ 178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:30.380819 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:30.380830 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:30.380838 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:46:30.381023 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 02:46:30.381112 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10085,"thread_start_us":9884,"threads_started":1}
I20260929 02:46:30.381666 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.367730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53192 (local address 127.0.97.2:39781)
0929 02:46:30.378516 (+ 10786us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.378527 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.379153 (+ 626us) server_negotiation.cc:407] Connection header received
0929 02:46:30.379242 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.379253 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.379480 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.379647 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.380784 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.380792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.380806 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.380836 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.380888 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.380909 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.381027 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.381242 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.381362 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10602,"thread_start_us":10469,"threads_started":1}
I20260929 02:46:30.382375 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.367304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39781 (local address 127.0.0.1:53192)
0929 02:46:30.378882 (+ 11578us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.378900 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:30.379006 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:30.379668 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:30.379672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:30.379693 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:30.380591 (+ 898us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:30.380595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:30.380610 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:30.380653 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:30.381076 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:30.381080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:30.381084 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 02:46:30.381874 (+ 790us) client_negotiation.cc:240] Negotiation successful
0929 02:46:30.381928 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11403,"thread_start_us":11237,"threads_started":1}
I20260929 02:46:30.382799 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.367389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.0.97.1:43519)
0929 02:46:30.377091 (+ 9702us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.377104 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.378158 (+ 1054us) server_negotiation.cc:407] Connection header received
0929 02:46:30.378274 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.378287 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.378604 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.378823 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.380524 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.380538 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.380585 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.380613 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.380671 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.380697 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.380844 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.382365 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.382464 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9510,"thread_start_us":9384,"threads_started":1}
I20260929 02:46:30.385300 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.379971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39841 (local address 127.0.0.1:44998)
0929 02:46:30.382095 (+ 2124us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.382111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:30.382189 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:30.383590 (+ 1401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:30.383594 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:30.383625 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:30.384356 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:30.384358 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:30.384377 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:30.384421 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:30.384942 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:30.384945 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:30.384948 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 02:46:30.385057 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 02:46:30.385100 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2025,"thread_start_us":2716,"threads_started":1}
I20260929 02:46:30.385953 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.379322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.0.97.3:39841)
0929 02:46:30.383215 (+ 3893us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.383222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.383247 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:46:30.383304 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.383309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.383444 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.383595 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.384627 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.384636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.384649 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.384670 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.384729 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.384747 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.384899 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.385621 (+ 722us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.385689 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3765,"thread_start_us":3626,"threads_started":1}
I20260929 02:46:30.387040 5172 tablet_service.cc:1467] Processing CreateTablet for tablet 05c727429f00497f9eeefe98b9ba12f0 (DEFAULT_TABLE table=test-table [id=99c23987aaa3484ca073575e1d2d067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:30.387575 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 05c727429f00497f9eeefe98b9ba12f0 (DEFAULT_TABLE table=test-table [id=99c23987aaa3484ca073575e1d2d067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:30.388494 5172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c727429f00497f9eeefe98b9ba12f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:30.388842 5303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c727429f00497f9eeefe98b9ba12f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:30.389163 5433 tablet_service.cc:1467] Processing CreateTablet for tablet 05c727429f00497f9eeefe98b9ba12f0 (DEFAULT_TABLE table=test-table [id=99c23987aaa3484ca073575e1d2d067c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:30.390189 5433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05c727429f00497f9eeefe98b9ba12f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:30.401726 5516 tablet_bootstrap.cc:492] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Bootstrap starting.
I20260929 02:46:30.404863 5516 tablet_bootstrap.cc:654] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:30.405915 5516 log.cc:824] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:30.408074 5518 tablet_bootstrap.cc:492] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: Bootstrap starting.
I20260929 02:46:30.409834 5517 tablet_bootstrap.cc:492] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Bootstrap starting.
I20260929 02:46:30.412691 5517 tablet_bootstrap.cc:654] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:30.412709 5518 tablet_bootstrap.cc:654] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:30.413866 5517 log.cc:824] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:30.414278 5518 log.cc:824] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:30.414867 5516 tablet_bootstrap.cc:492] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: No bootstrap required, opened a new log
I20260929 02:46:30.415208 5516 ts_tablet_manager.cc:1397] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260929 02:46:30.423285 5517 tablet_bootstrap.cc:492] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: No bootstrap required, opened a new log
I20260929 02:46:30.423581 5517 ts_tablet_manager.cc:1397] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260929 02:46:30.424276 5518 tablet_bootstrap.cc:492] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: No bootstrap required, opened a new log
I20260929 02:46:30.424535 5518 ts_tablet_manager.cc:1397] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20260929 02:46:30.426810 5516 raft_consensus.cc:348] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.427265 5516 raft_consensus.cc:374] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:30.427371 5516 raft_consensus.cc:729] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3263bd37704a61a593d4fa52bf7495, State: Initialized, Role: FOLLOWER
I20260929 02:46:30.428009 5516 consensus_queue.cc:260] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.431998 5518 raft_consensus.cc:348] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.432360 5518 raft_consensus.cc:374] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:30.432449 5518 raft_consensus.cc:729] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a3e396b154486f98316e3ebad16b99, State: Initialized, Role: FOLLOWER
I20260929 02:46:30.433084 5518 consensus_queue.cc:260] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.435411 5498 heartbeater.cc:503] Master 127.0.97.62:36685 was elected leader, sending a full tablet report...
I20260929 02:46:30.435690 5517 raft_consensus.cc:348] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.436120 5517 raft_consensus.cc:374] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:30.436200 5517 raft_consensus.cc:729] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a80310dc93a84ffd93e24b3e1c20634b, State: Initialized, Role: FOLLOWER
I20260929 02:46:30.436818 5517 consensus_queue.cc:260] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.445779 5516 ts_tablet_manager.cc:1428] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Time spent starting tablet: real 0.030s user 0.009s sys 0.013s
I20260929 02:46:30.450412 5237 heartbeater.cc:503] Master 127.0.97.62:36685 was elected leader, sending a full tablet report...
I20260929 02:46:30.450992 5368 heartbeater.cc:503] Master 127.0.97.62:36685 was elected leader, sending a full tablet report...
I20260929 02:46:30.451186 5518 ts_tablet_manager.cc:1428] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: Time spent starting tablet: real 0.026s user 0.007s sys 0.017s
I20260929 02:46:30.452651 5517 ts_tablet_manager.cc:1428] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Time spent starting tablet: real 0.029s user 0.019s sys 0.008s
W20260929 02:46:30.565312 5499 tablet.cc:2369] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:30.606079 5523 raft_consensus.cc:482] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:30.606449 5523 raft_consensus.cc:504] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.608105 5523 leader_election.cc:288] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a80310dc93a84ffd93e24b3e1c20634b (127.0.97.2:39781), 1d3263bd37704a61a593d4fa52bf7495 (127.0.97.3:39841)
I20260929 02:46:30.611640 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.608659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60425 (local address 127.0.97.2:39781)
0929 02:46:30.608880 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.608889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.609335 (+ 446us) server_negotiation.cc:407] Connection header received
0929 02:46:30.609536 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.609543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.609628 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.609735 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.610928 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.610937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.610945 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.610970 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.611013 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.611031 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.611153 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.611264 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.611345 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:46:30.611943 5525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.608495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39781 (local address 127.0.97.1:60425)
0929 02:46:30.609164 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.609185 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:30.609376 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:30.609862 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:30.609867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:30.609897 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:30.610683 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:30.610687 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:30.610707 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:30.610757 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:30.611142 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:30.611147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:30.611152 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 02:46:30.611317 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 02:46:30.611423 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":431,"threads_started":1}
I20260929 02:46:30.612046 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.609608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36253 (local address 127.0.97.3:39841)
0929 02:46:30.609799 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.609805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.609990 (+ 185us) server_negotiation.cc:407] Connection header received
0929 02:46:30.610171 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.610178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.610252 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.610357 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.611365 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.611370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.611374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.611392 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.611428 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.611451 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.611574 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.611743 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.611800 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:46:30.612311 5526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.609276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39841 (local address 127.0.97.1:36253)
0929 02:46:30.609898 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.609912 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:30.610035 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:30.610382 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:30.610385 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:30.610400 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:30.611168 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:46:30.611171 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:30.611187 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:46:30.611232 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:30.611596 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:30.611602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:30.611608 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:46:30.611774 (+ 166us) client_negotiation.cc:240] Negotiation successful
0929 02:46:30.611835 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":318,"threads_started":1}
I20260929 02:46:30.612982 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c727429f00497f9eeefe98b9ba12f0" candidate_uuid: "a4a3e396b154486f98316e3ebad16b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80310dc93a84ffd93e24b3e1c20634b" is_pre_election: true
I20260929 02:46:30.613194 5453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c727429f00497f9eeefe98b9ba12f0" candidate_uuid: "a4a3e396b154486f98316e3ebad16b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3263bd37704a61a593d4fa52bf7495" is_pre_election: true
I20260929 02:46:30.613483 5323 raft_consensus.cc:2454] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a3e396b154486f98316e3ebad16b99 in term 0.
I20260929 02:46:30.613692 5453 raft_consensus.cc:2454] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a3e396b154486f98316e3ebad16b99 in term 0.
I20260929 02:46:30.614115 5128 leader_election.cc:302] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a3e396b154486f98316e3ebad16b99, a80310dc93a84ffd93e24b3e1c20634b; no voters:
I20260929 02:46:30.614427 5523 raft_consensus.cc:2781] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:30.614549 5523 raft_consensus.cc:482] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:30.614671 5523 raft_consensus.cc:3037] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:30.616878 5523 raft_consensus.cc:504] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.617446 5523 leader_election.cc:288] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [CANDIDATE]: Term 1 election: Requested vote from peers a80310dc93a84ffd93e24b3e1c20634b (127.0.97.2:39781), 1d3263bd37704a61a593d4fa52bf7495 (127.0.97.3:39841)
I20260929 02:46:30.617918 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c727429f00497f9eeefe98b9ba12f0" candidate_uuid: "a4a3e396b154486f98316e3ebad16b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a80310dc93a84ffd93e24b3e1c20634b"
I20260929 02:46:30.618008 5453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05c727429f00497f9eeefe98b9ba12f0" candidate_uuid: "a4a3e396b154486f98316e3ebad16b99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3263bd37704a61a593d4fa52bf7495"
I20260929 02:46:30.618116 5323 raft_consensus.cc:3037] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:30.618232 5453 raft_consensus.cc:3037] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:30.620369 5453 raft_consensus.cc:2454] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a3e396b154486f98316e3ebad16b99 in term 1.
I20260929 02:46:30.620368 5323 raft_consensus.cc:2454] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a3e396b154486f98316e3ebad16b99 in term 1.
I20260929 02:46:30.620759 5128 leader_election.cc:302] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4a3e396b154486f98316e3ebad16b99, a80310dc93a84ffd93e24b3e1c20634b; no voters:
I20260929 02:46:30.621068 5523 raft_consensus.cc:2781] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:30.621575 5523 raft_consensus.cc:686] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 1 LEADER]: Becoming Leader. State: Replica: a4a3e396b154486f98316e3ebad16b99, State: Running, Role: LEADER
W20260929 02:46:30.621793 5369 tablet.cc:2369] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:30.622159 5523 consensus_queue.cc:237] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } }
I20260929 02:46:30.626585 5051 catalog_manager.cc:5495] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a3e396b154486f98316e3ebad16b99 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "a4a3e396b154486f98316e3ebad16b99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a3e396b154486f98316e3ebad16b99" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43519 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:30.687469 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.685701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.0.97.1:43519)
0929 02:46:30.685886 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.685891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.685908 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:46:30.685994 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.685997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.686072 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.686166 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.687016 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.687019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.687022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.687033 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.687056 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.687067 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.687149 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.687218 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.687259 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 02:46:30.690392 5238 tablet.cc:2369] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:30.695883 5323 raft_consensus.cc:1261] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 1 FOLLOWER]: Refusing update from remote peer a4a3e396b154486f98316e3ebad16b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:30.695885 5453 raft_consensus.cc:1261] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 1 FOLLOWER]: Refusing update from remote peer a4a3e396b154486f98316e3ebad16b99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:30.696882 5527 consensus_queue.cc:1035] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a80310dc93a84ffd93e24b3e1c20634b" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:30.697252 5523 consensus_queue.cc:1035] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3263bd37704a61a593d4fa52bf7495" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:30.724097 5051 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:30.724431 5051 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:30.727921 5051 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11944bdbc82247be9f0d748d8e2f25ae: Sending DeleteTablet for 3 replicas of tablet 05c727429f00497f9eeefe98b9ba12f0
I20260929 02:46:30.729107 5172 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c727429f00497f9eeefe98b9ba12f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:30 UTC) from {username='slave'} at 127.0.0.1:35350
I20260929 02:46:30.729236 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c727429f00497f9eeefe98b9ba12f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:30 UTC) from {username='slave'} at 127.0.0.1:53192
I20260929 02:46:30.730228 5433 tablet_service.cc:1514] Processing DeleteTablet for tablet 05c727429f00497f9eeefe98b9ba12f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:30 UTC) from {username='slave'} at 127.0.0.1:44998
I20260929 02:46:30.730758 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5110
I20260929 02:46:30.745029 5541 tablet_replica.cc:330] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99: stopping tablet replica
I20260929 02:46:30.745568 5541 raft_consensus.cc:2229] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:30.746349 5541 raft_consensus.cc:2258] T 05c727429f00497f9eeefe98b9ba12f0 P a4a3e396b154486f98316e3ebad16b99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:30.748118 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.732606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.0.97.1:43519)
0929 02:46:30.732833 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.732840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.743044 (+ 10204us) server_negotiation.cc:407] Connection header received
0929 02:46:30.743637 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.743645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.743722 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.743807 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.747188 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.747196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.747202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.747226 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.747270 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.747287 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.747405 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.747723 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.747802 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:46:30.748704 5542 tablet_replica.cc:330] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: stopping tablet replica
I20260929 02:46:30.749418 5232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:30.749353 5542 raft_consensus.cc:2229] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:30.750128 5542 raft_consensus.cc:2258] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:30.751160 5540 tablet_replica.cc:330] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: stopping tablet replica
I20260929 02:46:30.751705 5540 raft_consensus.cc:2229] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:30.752364 5540 raft_consensus.cc:2258] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:30.754204 5542 ts_tablet_manager.cc:1905] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:30.754900 5540 ts_tablet_manager.cc:1905] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:30.760835 5540 ts_tablet_manager.cc:1918] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:30.761047 5540 log.cc:1196] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/05c727429f00497f9eeefe98b9ba12f0
I20260929 02:46:30.761566 5540 ts_tablet_manager.cc:1939] T 05c727429f00497f9eeefe98b9ba12f0 P a80310dc93a84ffd93e24b3e1c20634b: Deleting consensus metadata
I20260929 02:46:30.761762 5542 ts_tablet_manager.cc:1918] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:30.761955 5542 log.cc:1196] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/05c727429f00497f9eeefe98b9ba12f0
I20260929 02:46:30.762494 5542 ts_tablet_manager.cc:1939] T 05c727429f00497f9eeefe98b9ba12f0 P 1d3263bd37704a61a593d4fa52bf7495: Deleting consensus metadata
I20260929 02:46:30.763543 5038 catalog_manager.cc:4841] TS a80310dc93a84ffd93e24b3e1c20634b (127.0.97.2:39781): tablet 05c727429f00497f9eeefe98b9ba12f0 (table test-table [id=99c23987aaa3484ca073575e1d2d067c]) successfully deleted
I20260929 02:46:30.764326 5038 catalog_manager.cc:4841] TS 1d3263bd37704a61a593d4fa52bf7495 (127.0.97.3:39841): tablet 05c727429f00497f9eeefe98b9ba12f0 (table test-table [id=99c23987aaa3484ca073575e1d2d067c]) successfully deleted
I20260929 02:46:30.849701 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5110
I20260929 02:46:30.867128 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5241
W20260929 02:46:30.867111 5036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43519 (error 108)
I20260929 02:46:30.869237 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.868729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35374)
0929 02:46:30.868910 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.869045 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 02:46:30.869542 5036 catalog_manager.cc:4584] TS a4a3e396b154486f98316e3ebad16b99 (127.0.97.1:43519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 05c727429f00497f9eeefe98b9ba12f0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
I20260929 02:46:30.870841 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.868386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.0.97.2:39781)
0929 02:46:30.868740 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.868745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.868759 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:46:30.868841 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.868846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.868951 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.869062 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.870129 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.870135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.870140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.870166 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.870203 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.870223 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.870379 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.870467 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.870528 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260929 02:46:30.871487 5363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:30.923131 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.922697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35384)
0929 02:46:30.922896 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.922990 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:46:30.924122 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.923803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35400)
0929 02:46:30.923939 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.924017 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 02:46:30.959990 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5241
I20260929 02:46:30.976218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5371
I20260929 02:46:30.979512 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.977039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.0.97.3:39841)
0929 02:46:30.977220 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:30.977227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:30.977311 (+ 84us) server_negotiation.cc:407] Connection header received
0929 02:46:30.977561 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:30.977565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:30.977691 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:30.977815 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:30.978818 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:30.978823 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:30.978827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:30.978842 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:30.978876 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:30.978889 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:30.979033 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:30.979116 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 02:46:30.979202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:46:30.980196 5493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:30.984691 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.984124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35404)
0929 02:46:30.984400 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.984503 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260929 02:46:30.985687 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:30.985360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35418)
0929 02:46:30.985491 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:30.985556 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:46:31.066279 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:31.065792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35420)
0929 02:46:31.066019 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:31.066125 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:46:31.067831 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:31.067072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43519 (local address 127.0.0.1:35434)
0929 02:46:31.067249 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:31.067327 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260929 02:46:31.069401 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5371
I20260929 02:46:31.085703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5020
I20260929 02:46:31.086932 5081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:31.194706 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5020
2026-09-29T02:46:31Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2770 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 02:46:31.235433 388 mini_oidc.cc:152] Starting JWKS server
I20260929 02:46:31.238126 388 webserver.cc:460] Webserver started at https://127.0.0.1:38515/ using document root <none> and password file <none>
I20260929 02:46:31.238469 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:46:31.239180 388 webserver.cc:460] Webserver started at http://127.0.0.1:41547/ using document root <none> and password file <none>
2026-09-29T02:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:31Z Disabled control of system clock
I20260929 02:46:31.265168 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45269
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38515/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/testchainca.cert with env {}
W20260929 02:46:31.588868 5554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:31.589190 5554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:31.589279 5554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:31.596467 5554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:31.596630 5554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:31.596700 5554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:31.596760 5554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:31.596796 5554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:46:31.596858 5554 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:46:31.596896 5554 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38515/jwks.json
I20260929 02:46:31.606927 5554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45269
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38515/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:31.608238 5554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:31.609563 5554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:31.621268 5559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:31.621316 5562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:31.621349 5560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:31.622884 5554 server_base.cc:1034] running on GCE node
I20260929 02:46:31.623765 5554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:31.625418 5554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:31.626693 5554 hybrid_clock.cc:648] HybridClock initialized: now 1790649991626643 us; error 61 us; skew 500 ppm
I20260929 02:46:31.627077 5554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:31.629492 5554 webserver.cc:460] Webserver started at http://127.0.97.62:35887/ using document root <none> and password file <none>
I20260929 02:46:31.630190 5554 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:31.630331 5554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:31.630631 5554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:31.633663 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "5190c05f5060420993384acfa9cc39f0"
format_stamp: "Formatted at 2026-09-29 02:46:31 on dist-test-slave-6tjh"
server_key: "514ce4ae663d5774de2cde493bf68a04"
server_key_iv: "a0bbb21873f1fe5814e651b4e37de328"
server_key_version: "encryptionkey@0"
I20260929 02:46:31.634465 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "5190c05f5060420993384acfa9cc39f0"
format_stamp: "Formatted at 2026-09-29 02:46:31 on dist-test-slave-6tjh"
server_key: "514ce4ae663d5774de2cde493bf68a04"
server_key_iv: "a0bbb21873f1fe5814e651b4e37de328"
server_key_version: "encryptionkey@0"
I20260929 02:46:31.640033 5554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 02:46:31.643744 5568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:31.645545 5554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:31.645727 5554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "5190c05f5060420993384acfa9cc39f0"
format_stamp: "Formatted at 2026-09-29 02:46:31 on dist-test-slave-6tjh"
server_key: "514ce4ae663d5774de2cde493bf68a04"
server_key_iv: "a0bbb21873f1fe5814e651b4e37de328"
server_key_version: "encryptionkey@0"
I20260929 02:46:31.645960 5554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:31.663573 5554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:31.664427 5554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:31.664745 5554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:31.682006 5554 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45269
I20260929 02:46:31.682024 5619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45269 every 8 connection(s)
I20260929 02:46:31.683432 5554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:31.687219 5620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:31.693496 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5554
I20260929 02:46:31.693715 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:31.694523 388 external_mini_cluster.cc:1428] Setting key 7b66ce844c177d5ef406f46311dca02e
I20260929 02:46:31.698680 5620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0: Bootstrap starting.
I20260929 02:46:31.701907 5620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:31.702915 5620 log.cc:824] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:31.705690 5620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0: No bootstrap required, opened a new log
I20260929 02:46:31.708235 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:31.696570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.0.97.62:45269)
0929 02:46:31.697572 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:31.697584 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:31.697625 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:46:31.699401 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:31.699424 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:31.699890 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:31.700306 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:31.700913 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:31.702256 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:31.703671 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:31.704508 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:31.704765 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:31.705924 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:31.705932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:31.705947 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:31.706485 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:31.706559 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:31.706579 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:31.706713 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:31.706929 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 02:46:31.707251 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":329,"threads_started":1}
I20260929 02:46:31.712304 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:46:31.714177 5620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5190c05f5060420993384acfa9cc39f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 02:46:31.714576 5620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:31.714696 5620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5190c05f5060420993384acfa9cc39f0, State: Initialized, Role: FOLLOWER
I20260929 02:46:31.715363 5620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5190c05f5060420993384acfa9cc39f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 02:46:31.715598 5620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:31.715742 5620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:31.715927 5620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:31.718142 5620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5190c05f5060420993384acfa9cc39f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 02:46:31.718650 5620 leader_election.cc:302] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5190c05f5060420993384acfa9cc39f0; no voters:
I20260929 02:46:31.719257 5620 leader_election.cc:288] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:31.719465 5625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:31.720388 5625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [term 1 LEADER]: Becoming Leader. State: Replica: 5190c05f5060420993384acfa9cc39f0, State: Running, Role: LEADER
I20260929 02:46:31.720904 5625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5190c05f5060420993384acfa9cc39f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } }
I20260929 02:46:31.721673 5620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:31.723558 5626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5190c05f5060420993384acfa9cc39f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5190c05f5060420993384acfa9cc39f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } } }
I20260929 02:46:31.723954 5626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:31.730463 5630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:31.731503 5627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5190c05f5060420993384acfa9cc39f0. Latest consensus state: current_term: 1 leader_uuid: "5190c05f5060420993384acfa9cc39f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5190c05f5060420993384acfa9cc39f0" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45269 } } }
I20260929 02:46:31.732020 5627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:31.737916 5630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:31.747179 5630 catalog_manager.cc:1347] Generated new cluster ID: fb08984a418b494c8874c9f381c4acdb
I20260929 02:46:31.747313 5630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:31.774780 5630 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:31.776137 5630 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:31.785660 5630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5190c05f5060420993384acfa9cc39f0: Generated new TSK 0
I20260929 02:46:31.786504 5630 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:46:31.830933 5650 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 02:46:31.831130 5650 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:46:31.832252 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:31.810668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.0.97.62:45269)
0929 02:46:31.810865 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:31.810872 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:31.811402 (+ 530us) server_negotiation.cc:407] Connection header received
0929 02:46:31.811720 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:31.811726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:31.811820 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:31.811994 (+ 174us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:46:31.812514 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:31.813668 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:31.815286 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:31.815761 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:31.815902 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:31.816434 (+ 532us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:46:31.831579 (+ 15145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:46:31.831968 (+ 389us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 02:46:31.832388 5623 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33552: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 02:46:31.833811 5649 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:31.810542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45269 (local address 127.0.0.1:33552)
0929 02:46:31.811302 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:31.811318 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:31.811475 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:31.812018 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:31.812021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:31.812031 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:46:31.812366 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:31.812374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:31.813847 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:31.813852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:31.815009 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:31.815021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:31.815902 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:31.815973 (+ 71us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:31.816107 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:31.816166 (+ 59us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:46:31.831898 (+ 15732us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 02:46:31.832253 (+ 355us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:45269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":427,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:45269: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38515/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 02:46:31.835714 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5554
I20260929 02:46:31.836822 5615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:31.930310 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5554
2026-09-29T02:46:31Z chronyd exiting
I20260929 02:46:31.980139 388 test_util.cc:175] -----------------------------------------------
I20260929 02:46:31.980274 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790649949495923-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (749 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 02:46:31.983863 388 mini_oidc.cc:152] Starting JWKS server
I20260929 02:46:31.985728 388 webserver.cc:460] Webserver started at https://127.0.0.1:43851/ using document root <none> and password file <none>
I20260929 02:46:31.985905 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:46:31.986481 388 webserver.cc:460] Webserver started at http://127.0.0.1:41875/ using document root <none> and password file <none>
2026-09-29T02:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:31Z Disabled control of system clock
I20260929 02:46:32.013113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36601
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:41781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36601
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43851/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/testchainca.cert with env {}
W20260929 02:46:32.326992 5657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:32.327340 5657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:32.327507 5657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:32.334919 5657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:32.335083 5657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:32.335180 5657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:32.335295 5657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:32.335383 5657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:46:32.335472 5657 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:46:32.335551 5657 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43851/jwks.json
I20260929 02:46:32.345796 5657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43851/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:32.347223 5657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:32.348634 5657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:32.361429 5663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:32.361382 5665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:32.361759 5662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:32.362597 5657 server_base.cc:1034] running on GCE node
I20260929 02:46:32.363463 5657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:32.365105 5657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:32.366374 5657 hybrid_clock.cc:648] HybridClock initialized: now 1790649992366339 us; error 47 us; skew 500 ppm
I20260929 02:46:32.366765 5657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:32.373353 5657 webserver.cc:460] Webserver started at http://127.0.97.62:42707/ using document root <none> and password file <none>
I20260929 02:46:32.374188 5657 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:32.374347 5657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:32.374701 5657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:32.377907 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "3b964917260044b8947960f79d7de4cd"
format_stamp: "Formatted at 2026-09-29 02:46:32 on dist-test-slave-6tjh"
server_key: "4abd76079b369f6d0992346e63bd5c83"
server_key_iv: "100db7c1f48ebb443a5c19c8f95c6cdb"
server_key_version: "encryptionkey@0"
I20260929 02:46:32.378762 5657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "3b964917260044b8947960f79d7de4cd"
format_stamp: "Formatted at 2026-09-29 02:46:32 on dist-test-slave-6tjh"
server_key: "4abd76079b369f6d0992346e63bd5c83"
server_key_iv: "100db7c1f48ebb443a5c19c8f95c6cdb"
server_key_version: "encryptionkey@0"
I20260929 02:46:32.384184 5657 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:46:32.387722 5671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:32.389084 5657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:32.389268 5657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "3b964917260044b8947960f79d7de4cd"
format_stamp: "Formatted at 2026-09-29 02:46:32 on dist-test-slave-6tjh"
server_key: "4abd76079b369f6d0992346e63bd5c83"
server_key_iv: "100db7c1f48ebb443a5c19c8f95c6cdb"
server_key_version: "encryptionkey@0"
I20260929 02:46:32.389506 5657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:32.427918 5657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:32.428874 5657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:32.429170 5657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:32.447301 5657 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36601
I20260929 02:46:32.447309 5722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36601 every 8 connection(s)
I20260929 02:46:32.448699 5657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:32.452334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5657
I20260929 02:46:32.452520 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:32.452665 5723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:32.453351 388 external_mini_cluster.cc:1428] Setting key 60975c2db11cb54723b81e44499776a9
I20260929 02:46:32.465207 5723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd: Bootstrap starting.
I20260929 02:46:32.467319 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:32.455278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49346 (local address 127.0.97.62:36601)
0929 02:46:32.456615 (+ 1337us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:32.456628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:32.456674 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:46:32.458979 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:32.459001 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:32.459486 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:32.459889 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:32.460435 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:32.461724 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:32.462997 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:32.463676 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:32.463863 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:32.465082 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:32.465089 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:32.465102 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:32.465683 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:32.465770 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:32.465791 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:32.465928 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:32.466099 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 02:46:32.466401 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":425,"threads_started":1}
I20260929 02:46:32.468775 5723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:32.469982 5723 log.cc:824] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:32.471413 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:46:32.472924 5723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd: No bootstrap required, opened a new log
I20260929 02:46:32.480015 5723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b964917260044b8947960f79d7de4cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36601 } }
I20260929 02:46:32.480406 5723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:32.480531 5723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b964917260044b8947960f79d7de4cd, State: Initialized, Role: FOLLOWER
I20260929 02:46:32.481099 5723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b964917260044b8947960f79d7de4cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36601 } }
I20260929 02:46:32.481341 5723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:32.481485 5723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:32.481652 5723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:32.483781 5723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b964917260044b8947960f79d7de4cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36601 } }
I20260929 02:46:32.484297 5723 leader_election.cc:302] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b964917260044b8947960f79d7de4cd; no voters:
I20260929 02:46:32.484937 5723 leader_election.cc:288] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:32.485061 5728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:32.485733 5728 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [term 1 LEADER]: Becoming Leader. State: Replica: 3b964917260044b8947960f79d7de4cd, State: Running, Role: LEADER
I20260929 02:46:32.486269 5728 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b964917260044b8947960f79d7de4cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36601 } }
I20260929 02:46:32.486980 5723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:32.488965 5729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b964917260044b8947960f79d7de4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b964917260044b8947960f79d7de4cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36601 } } }
I20260929 02:46:32.489396 5729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:32.490092 5730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b964917260044b8947960f79d7de4cd. Latest consensus state: current_term: 1 leader_uuid: "3b964917260044b8947960f79d7de4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b964917260044b8947960f79d7de4cd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36601 } } }
I20260929 02:46:32.490878 5730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:32.491050 5736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:32.495913 5736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:32.505559 5736 catalog_manager.cc:1347] Generated new cluster ID: c651b37074a3451294bb835ffa7a94a1
I20260929 02:46:32.505699 5736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:32.525312 5736 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:32.527212 5736 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:32.537606 5736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b964917260044b8947960f79d7de4cd: Generated new TSK 0
I20260929 02:46:32.538514 5736 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 02:46:32.579802 5753 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 02:46:32.579946 5753 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:46:32.581085 5726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:32.560696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.0.97.62:36601)
0929 02:46:32.560926 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:32.560933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:32.561309 (+ 376us) server_negotiation.cc:407] Connection header received
0929 02:46:32.561543 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:32.561547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:32.561617 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:32.561751 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:46:32.562308 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:32.563018 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:32.565061 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:32.565531 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:32.565629 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:32.566313 (+ 684us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:46:32.580470 (+ 14157us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43851/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:46:32.580856 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49362: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43851/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":63}
W20260929 02:46:32.581238 5726 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49362: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43851/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 02:46:32.581516 5752 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:32.560521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36601 (local address 127.0.0.1:49362)
0929 02:46:32.561190 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:32.561211 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:32.561371 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:32.561796 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:32.561802 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:32.561815 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:46:32.562133 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:32.562146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:32.563149 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:32.563155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:32.564906 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:32.564922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:32.565654 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:32.565704 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:32.565920 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:32.565968 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:46:32.580819 (+ 14851us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43851/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 02:46:32.581116 (+ 297us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:36601: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43851/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":321,"threads_started":1}
I20260929 02:46:32.582849 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5657
I20260929 02:46:32.584007 5718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:32.676721 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5657
2026-09-29T02:46:32Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (749 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 02:46:32.733826 388 mini_oidc.cc:152] Starting JWKS server
I20260929 02:46:32.735684 388 webserver.cc:460] Webserver started at https://127.0.0.1:38673/ using document root <none> and password file <none>
I20260929 02:46:32.735883 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 02:46:32.736457 388 webserver.cc:460] Webserver started at http://127.0.0.1:40135/ using document root <none> and password file <none>
2026-09-29T02:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:32Z Disabled control of system clock
I20260929 02:46:32.763351 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41063
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41063
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38673/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:46:33.091704 5760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:33.092283 5760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:33.092518 5760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:33.104328 5760 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:33.104489 5760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:33.104532 5760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:33.104638 5760 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:33.104704 5760 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:46:33.104786 5760 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:46:33.104861 5760 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38673/jwks.json
I20260929 02:46:33.119405 5760 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38673/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:33.121192 5760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:33.123909 5760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:33.139309 5768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:33.139303 5766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:33.140003 5765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:33.140523 5760 server_base.cc:1034] running on GCE node
I20260929 02:46:33.141371 5760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:33.142807 5760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:33.144026 5760 hybrid_clock.cc:648] HybridClock initialized: now 1790649993143978 us; error 67 us; skew 500 ppm
I20260929 02:46:33.144371 5760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:33.147163 5760 webserver.cc:460] Webserver started at http://127.0.97.62:40689/ using document root <none> and password file <none>
I20260929 02:46:33.147854 5760 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:33.147976 5760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:33.148315 5760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:33.151326 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "5276c1c6c7564d4b801646e1b2c38787"
format_stamp: "Formatted at 2026-09-29 02:46:33 on dist-test-slave-6tjh"
server_key: "ca53349706bf53281cf68eb20fcc2b5a"
server_key_iv: "0ddf32b1e6519815eb99ae12d518ea49"
server_key_version: "encryptionkey@0"
I20260929 02:46:33.152125 5760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "5276c1c6c7564d4b801646e1b2c38787"
format_stamp: "Formatted at 2026-09-29 02:46:33 on dist-test-slave-6tjh"
server_key: "ca53349706bf53281cf68eb20fcc2b5a"
server_key_iv: "0ddf32b1e6519815eb99ae12d518ea49"
server_key_version: "encryptionkey@0"
I20260929 02:46:33.157449 5760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:46:33.161037 5774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:33.162493 5760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:46:33.162678 5760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "5276c1c6c7564d4b801646e1b2c38787"
format_stamp: "Formatted at 2026-09-29 02:46:33 on dist-test-slave-6tjh"
server_key: "ca53349706bf53281cf68eb20fcc2b5a"
server_key_iv: "0ddf32b1e6519815eb99ae12d518ea49"
server_key_version: "encryptionkey@0"
I20260929 02:46:33.162899 5760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:33.185923 5760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:33.186832 5760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:33.187130 5760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:33.205452 5760 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41063
I20260929 02:46:33.205470 5825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41063 every 8 connection(s)
I20260929 02:46:33.206882 5760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:33.210577 5826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:33.212139 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5760
I20260929 02:46:33.212363 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:33.213160 388 external_mini_cluster.cc:1428] Setting key e0791ebd2c95790236dca49825e60170
I20260929 02:46:33.224323 5826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787: Bootstrap starting.
I20260929 02:46:33.226784 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:33.215151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33544 (local address 127.0.97.62:41063)
0929 02:46:33.216148 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:33.216158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:33.216200 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:46:33.217749 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:33.217774 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:33.218234 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:33.218621 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:33.219183 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.220496 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.222421 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.223064 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.223189 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:33.224448 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:33.224460 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:33.224482 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:33.225108 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:33.225183 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:33.225207 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:33.225298 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:33.225520 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 02:46:33.225874 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":353,"threads_started":1}
I20260929 02:46:33.227677 5826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:33.228768 5826 log.cc:824] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:33.230676 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:46:33.231700 5826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787: No bootstrap required, opened a new log
I20260929 02:46:33.239795 5826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5276c1c6c7564d4b801646e1b2c38787" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41063 } }
I20260929 02:46:33.240221 5826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:33.240339 5826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5276c1c6c7564d4b801646e1b2c38787, State: Initialized, Role: FOLLOWER
I20260929 02:46:33.240991 5826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5276c1c6c7564d4b801646e1b2c38787" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41063 } }
I20260929 02:46:33.241225 5826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:33.241379 5826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:33.241554 5826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:33.244179 5826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5276c1c6c7564d4b801646e1b2c38787" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41063 } }
I20260929 02:46:33.244752 5826 leader_election.cc:302] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5276c1c6c7564d4b801646e1b2c38787; no voters:
I20260929 02:46:33.245311 5826 leader_election.cc:288] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:33.245476 5831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:33.246203 5831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [term 1 LEADER]: Becoming Leader. State: Replica: 5276c1c6c7564d4b801646e1b2c38787, State: Running, Role: LEADER
I20260929 02:46:33.246797 5831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5276c1c6c7564d4b801646e1b2c38787" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41063 } }
I20260929 02:46:33.247433 5826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:33.249629 5832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5276c1c6c7564d4b801646e1b2c38787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5276c1c6c7564d4b801646e1b2c38787" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41063 } } }
I20260929 02:46:33.250499 5832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:33.250113 5833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5276c1c6c7564d4b801646e1b2c38787. Latest consensus state: current_term: 1 leader_uuid: "5276c1c6c7564d4b801646e1b2c38787" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5276c1c6c7564d4b801646e1b2c38787" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41063 } } }
I20260929 02:46:33.250906 5833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:33.252046 5839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:33.257745 5839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:33.266196 5839 catalog_manager.cc:1347] Generated new cluster ID: 9892d253d4914c83bc58f7fa38851aac
I20260929 02:46:33.266325 5839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:33.277289 5839 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:33.279193 5839 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:33.296516 5839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5276c1c6c7564d4b801646e1b2c38787: Generated new TSK 0
I20260929 02:46:33.297662 5839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:46:33.322082 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:33.313189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.0.97.62:41063)
0929 02:46:33.313392 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:33.313399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:33.313897 (+ 498us) server_negotiation.cc:407] Connection header received
0929 02:46:33.314106 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:33.314112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:33.314214 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:33.314403 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:46:33.314851 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.316010 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.317576 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.318297 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.318444 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:33.319005 (+ 561us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:46:33.321413 (+ 2408us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:46:33.321752 (+ 339us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33546: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 02:46:33.322221 5829 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33546: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 02:46:33.323874 5855 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:33.313106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41063 (local address 127.0.0.1:33546)
0929 02:46:33.313782 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:33.313799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:33.313928 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:33.314417 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:33.314421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:33.314431 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:46:33.314700 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:33.314709 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.316199 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.316203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:33.317411 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:33.317425 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.318452 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.318520 (+ 68us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:33.318673 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:33.318715 (+ 42us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:46:33.321633 (+ 2918us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 02:46:33.323451 (+ 1818us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41063: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38673/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":1495,"thread_start_us":397,"threads_started":1}
I20260929 02:46:33.325160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5760
I20260929 02:46:33.326424 5821 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 02:46:33.337761 5856 webserver.cc:533] Webserver: sslize failed:
E20260929 02:46:33.337921 5856 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 02:46:33.419948 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5760
2026-09-29T02:46:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (741 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T02:46:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:33Z Disabled control of system clock
I20260929 02:46:33.501281 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44049
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44049
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/testchainca.cert with env {}
W20260929 02:46:33.826308 5861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:33.826608 5861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:33.826757 5861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:33.834002 5861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:33.834118 5861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:33.834187 5861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:33.834280 5861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:33.834368 5861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:46:33.834482 5861 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 02:46:33.834561 5861 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 02:46:33.844488 5861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44049
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:33.845914 5861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:33.847296 5861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:33.859391 5866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:33.859318 5869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:33.859345 5867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:33.860472 5861 server_base.cc:1034] running on GCE node
I20260929 02:46:33.861332 5861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:33.862754 5861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:33.864005 5861 hybrid_clock.cc:648] HybridClock initialized: now 1790649993863955 us; error 46 us; skew 500 ppm
I20260929 02:46:33.864322 5861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:46:33.866590 5861 webserver.cc:460] Webserver started at http://127.0.97.62:35617/ using document root <none> and password file <none>
I20260929 02:46:33.867234 5861 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:33.867323 5861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:33.867569 5861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:33.870523 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "d24ea70ed77341f3b508a8077429d23b"
format_stamp: "Formatted at 2026-09-29 02:46:33 on dist-test-slave-6tjh"
server_key: "c029e40c2e806469e0cbadb3b0185700"
server_key_iv: "4b88ae61de6ccf46a05c4727827eaf67"
server_key_version: "encryptionkey@0"
I20260929 02:46:33.871282 5861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "d24ea70ed77341f3b508a8077429d23b"
format_stamp: "Formatted at 2026-09-29 02:46:33 on dist-test-slave-6tjh"
server_key: "c029e40c2e806469e0cbadb3b0185700"
server_key_iv: "4b88ae61de6ccf46a05c4727827eaf67"
server_key_version: "encryptionkey@0"
I20260929 02:46:33.876420 5861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:46:33.880003 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:33.881520 5861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:33.881709 5861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "d24ea70ed77341f3b508a8077429d23b"
format_stamp: "Formatted at 2026-09-29 02:46:33 on dist-test-slave-6tjh"
server_key: "c029e40c2e806469e0cbadb3b0185700"
server_key_iv: "4b88ae61de6ccf46a05c4727827eaf67"
server_key_version: "encryptionkey@0"
I20260929 02:46:33.881947 5861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:33.896065 5861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:33.897015 5861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:33.897323 5861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:33.916149 5861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44049
I20260929 02:46:33.916168 5926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44049 every 8 connection(s)
I20260929 02:46:33.917608 5861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:33.920132 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5861
I20260929 02:46:33.920341 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:33.921057 388 external_mini_cluster.cc:1428] Setting key ea03ce2604aa4e43cae187999a327d2a
I20260929 02:46:33.921401 5927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:33.934516 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b: Bootstrap starting.
I20260929 02:46:33.934749 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:33.922967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45290 (local address 127.0.97.62:44049)
0929 02:46:33.923953 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:33.923967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:33.924008 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:46:33.925752 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:33.925773 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:33.926211 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:33.926539 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:33.927144 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.928486 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.930257 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:33.931005 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:33.931179 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:33.932416 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:33.932424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:33.932437 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:46:33.933045 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:33.933119 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:46:33.933138 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:33.933230 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:33.933453 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 02:46:33.933804 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":350,"threads_started":1}
I20260929 02:46:33.937757 5927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:33.938400 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:46:33.938848 5927 log.cc:824] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:33.942018 5927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b: No bootstrap required, opened a new log
I20260929 02:46:33.949879 5927 raft_consensus.cc:348] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24ea70ed77341f3b508a8077429d23b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260929 02:46:33.950270 5927 raft_consensus.cc:374] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:33.950371 5927 raft_consensus.cc:729] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24ea70ed77341f3b508a8077429d23b, State: Initialized, Role: FOLLOWER
I20260929 02:46:33.950944 5927 consensus_queue.cc:260] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24ea70ed77341f3b508a8077429d23b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260929 02:46:33.951180 5927 raft_consensus.cc:388] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:33.951327 5927 raft_consensus.cc:482] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:33.951483 5927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:33.953756 5927 raft_consensus.cc:504] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24ea70ed77341f3b508a8077429d23b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260929 02:46:33.954298 5927 leader_election.cc:302] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d24ea70ed77341f3b508a8077429d23b; no voters:
I20260929 02:46:33.955032 5927 leader_election.cc:288] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:33.955176 5932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:33.955914 5932 raft_consensus.cc:686] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [term 1 LEADER]: Becoming Leader. State: Replica: d24ea70ed77341f3b508a8077429d23b, State: Running, Role: LEADER
I20260929 02:46:33.956485 5932 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24ea70ed77341f3b508a8077429d23b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } }
I20260929 02:46:33.957432 5927 sys_catalog.cc:564] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:33.960112 5933 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d24ea70ed77341f3b508a8077429d23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24ea70ed77341f3b508a8077429d23b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } } }
I20260929 02:46:33.960173 5934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24ea70ed77341f3b508a8077429d23b. Latest consensus state: current_term: 1 leader_uuid: "d24ea70ed77341f3b508a8077429d23b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24ea70ed77341f3b508a8077429d23b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44049 } } }
I20260929 02:46:33.960619 5934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:33.960616 5933 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:33.973165 5941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:33.978708 5941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:33.988077 5941 catalog_manager.cc:1347] Generated new cluster ID: f5896959ed794e158bfbfd172cf4df29
I20260929 02:46:33.988212 5941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:34.016112 5941 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:34.017494 5941 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:34.029168 5941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d24ea70ed77341f3b508a8077429d23b: Generated new TSK 0
I20260929 02:46:34.030076 5941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:46:34.072135 5930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:34.044767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.0.97.62:44049)
0929 02:46:34.044999 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:34.045007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:34.045365 (+ 358us) server_negotiation.cc:407] Connection header received
0929 02:46:34.045591 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:34.045595 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:34.045669 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:34.045829 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0929 02:46:34.046348 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:34.047073 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:34.049224 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:34.049661 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:34.049754 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:34.050462 (+ 708us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 02:46:34.071544 (+ 21082us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:46:34.071898 (+ 354us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45306: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 02:46:34.072307 5930 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45306: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 02:46:34.072829 5956 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:34.044584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:44049 (local address 127.0.0.1:45306)
0929 02:46:34.045244 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:34.045265 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:34.045429 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:34.045862 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:34.045868 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:34.045882 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 02:46:34.046197 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:34.046210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:34.047215 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:34.047222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:34.049083 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:34.049098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:34.049844 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:34.049941 (+ 97us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:34.050159 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:34.050216 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 02:46:34.071855 (+ 21639us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 02:46:34.072200 (+ 345us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:44049: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":331,"threads_started":1}
I20260929 02:46:34.074105 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5861
I20260929 02:46:34.075418 5922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:34.169502 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5861
2026-09-29T02:46:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (728 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (410 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (413 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:35 dist-test-slave-6tjh krb5kdc[5971](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:35 dist-test-slave-6tjh krb5kdc[5971](info): set up 2 sockets
Sep 29 02:46:35 dist-test-slave-6tjh krb5kdc[5971](info): commencing operation
krb5kdc: starting...
W20260929 02:46:36.097319 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:36 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:36.310627 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39391
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:36.630347 5984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:36.630702 5984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:36.630867 5984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:36.638352 5984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:36.638507 5984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:36.638619 5984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:36.638720 5984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:36.638805 5984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:36.648765 5984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:36.650192 5984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:36.651563 5984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:36.663743 5990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:36.663780 5992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:36.663808 5989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:36.664619 5984 server_base.cc:1034] running on GCE node
I20260929 02:46:36.665397 5984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:36.666890 5984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:36.668100 5984 hybrid_clock.cc:648] HybridClock initialized: now 1790649996668027 us; error 68 us; skew 500 ppm
I20260929 02:46:36.668465 5984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:36 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:36.671743 5984 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:36.673205 5984 webserver.cc:460] Webserver started at http://127.0.97.62:40889/ using document root <none> and password file <none>
I20260929 02:46:36.673869 5984 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:36.673993 5984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:36.674309 5984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:36.677285 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "ed6cde8067964a558ea55296e442ca69"
format_stamp: "Formatted at 2026-09-29 02:46:36 on dist-test-slave-6tjh"
server_key: "4cd978f0c68def97007c94d45d7a08e3"
server_key_iv: "10f1490d00eec92b3e7a2ea86c808ea9"
server_key_version: "encryptionkey@0"
I20260929 02:46:36.678040 5984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "ed6cde8067964a558ea55296e442ca69"
format_stamp: "Formatted at 2026-09-29 02:46:36 on dist-test-slave-6tjh"
server_key: "4cd978f0c68def97007c94d45d7a08e3"
server_key_iv: "10f1490d00eec92b3e7a2ea86c808ea9"
server_key_version: "encryptionkey@0"
I20260929 02:46:36.683192 5984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:46:36.686602 5999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:36.687919 5984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:46:36.688102 5984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "ed6cde8067964a558ea55296e442ca69"
format_stamp: "Formatted at 2026-09-29 02:46:36 on dist-test-slave-6tjh"
server_key: "4cd978f0c68def97007c94d45d7a08e3"
server_key_iv: "10f1490d00eec92b3e7a2ea86c808ea9"
server_key_version: "encryptionkey@0"
I20260929 02:46:36.688308 5984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:36.720048 5984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:36.724767 5984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:36.725065 5984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:36.743839 5984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39391
I20260929 02:46:36.743875 6050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39391 every 8 connection(s)
I20260929 02:46:36.745294 5984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:36.749357 6051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:36.751830 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 5984
I20260929 02:46:36.752031 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:36.752800 388 external_mini_cluster.cc:1428] Setting key 66f352daeca7c5bd2a56befe775022c9
I20260929 02:46:36.762081 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69: Bootstrap starting.
I20260929 02:46:36.765939 6051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:36.767366 6051 log.cc:824] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:46:36 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:36.770516 6051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69: No bootstrap required, opened a new log
I20260929 02:46:36.776770 6054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:36.754956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.0.97.62:39391)
0929 02:46:36.755781 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:36.755794 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:36.755835 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:46:36.757579 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:36.757603 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:36.758023 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:36.758386 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:36.760085 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:36.761474 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:36.763185 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:36.763910 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:36.764058 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:36.769174 (+ 5116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:36.769183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:36.769203 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:36.769238 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:36.772482 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:36.773284 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:36.773302 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:36.773309 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:36.773477 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:36.773879 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:36.773885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:36.773893 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:36.774442 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:36.774777 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:36.775232 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 02:46:36.775657 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":410,"threads_started":1}
I20260929 02:46:36.778668 6051 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6cde8067964a558ea55296e442ca69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260929 02:46:36.779148 6051 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:36.779256 6051 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed6cde8067964a558ea55296e442ca69, State: Initialized, Role: FOLLOWER
I20260929 02:46:36.779908 6051 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6cde8067964a558ea55296e442ca69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260929 02:46:36.780176 6051 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:36.780272 6051 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:36.780385 6051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:36.782977 6051 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6cde8067964a558ea55296e442ca69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260929 02:46:36.783408 6051 leader_election.cc:302] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed6cde8067964a558ea55296e442ca69; no voters:
I20260929 02:46:36.783931 6051 leader_election.cc:288] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:36.784106 6056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:36.784945 6056 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [term 1 LEADER]: Becoming Leader. State: Replica: ed6cde8067964a558ea55296e442ca69, State: Running, Role: LEADER
I20260929 02:46:36.785462 6056 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6cde8067964a558ea55296e442ca69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } }
I20260929 02:46:36.786163 6051 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:36.788409 6058 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed6cde8067964a558ea55296e442ca69. Latest consensus state: current_term: 1 leader_uuid: "ed6cde8067964a558ea55296e442ca69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6cde8067964a558ea55296e442ca69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } } }
I20260929 02:46:36.789470 6058 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:36.789326 6057 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed6cde8067964a558ea55296e442ca69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed6cde8067964a558ea55296e442ca69" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39391 } } }
I20260929 02:46:36.789629 6057 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:36.791082 6064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:36.795595 6064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:36.804414 6064 catalog_manager.cc:1347] Generated new cluster ID: c16f00edf2e146dcbada9a42551971f1
I20260929 02:46:36.804580 6064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:36.818567 6064 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:36.820634 6064 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:36.832216 6064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed6cde8067964a558ea55296e442ca69: Generated new TSK 0
I20260929 02:46:36.833379 6064 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:46:36.885845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39391
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:37.215701 6077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:37.215991 6077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:37.216082 6077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:37.223241 6077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:37.223367 6077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:37.223479 6077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:37.233414 6077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:37.234769 6077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:37.236042 6077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:37.249217 6083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:37.249244 6085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:37.249159 6082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:37.250240 6077 server_base.cc:1034] running on GCE node
I20260929 02:46:37.250891 6077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:37.251886 6077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:37.253145 6077 hybrid_clock.cc:648] HybridClock initialized: now 1790649997253094 us; error 35 us; skew 500 ppm
I20260929 02:46:37.253526 6077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:37 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:37.257014 6077 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:46:37.258706 6077 webserver.cc:460] Webserver started at http://127.0.97.1:37445/ using document root <none> and password file <none>
I20260929 02:46:37.259454 6077 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:37.259611 6077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:37.259941 6077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:37.263171 6077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "46c7ae7570ee4b1f8d693a80df73a0c9"
format_stamp: "Formatted at 2026-09-29 02:46:37 on dist-test-slave-6tjh"
server_key: "b9a977f6c27ba7e3a89135acf3e186cd"
server_key_iv: "a110cbc112cfaae61535fb891de7a9da"
server_key_version: "encryptionkey@0"
I20260929 02:46:37.264007 6077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "46c7ae7570ee4b1f8d693a80df73a0c9"
format_stamp: "Formatted at 2026-09-29 02:46:37 on dist-test-slave-6tjh"
server_key: "b9a977f6c27ba7e3a89135acf3e186cd"
server_key_iv: "a110cbc112cfaae61535fb891de7a9da"
server_key_version: "encryptionkey@0"
I20260929 02:46:37.269805 6077 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 02:46:37.273854 6092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:37.275337 6077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:46:37.275543 6077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "46c7ae7570ee4b1f8d693a80df73a0c9"
format_stamp: "Formatted at 2026-09-29 02:46:37 on dist-test-slave-6tjh"
server_key: "b9a977f6c27ba7e3a89135acf3e186cd"
server_key_iv: "a110cbc112cfaae61535fb891de7a9da"
server_key_version: "encryptionkey@0"
I20260929 02:46:37.275784 6077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:37.300149 6077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:37.304370 6077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:37.304726 6077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:37.305714 6077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:37.307090 6077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:37.307195 6077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:37.307338 6077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:37.307410 6077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:37.341377 6077 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36331
I20260929 02:46:37.341464 6204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36331 every 8 connection(s)
I20260929 02:46:37.342659 6077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:37.346964 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 6077
I20260929 02:46:37.347167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:37.348004 388 external_mini_cluster.cc:1428] Setting key 93835ddce8518dc982bb1f86d9cbace7
Sep 29 02:46:37 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:37.366967 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:37.345342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33757 (local address 127.0.97.62:39391)
0929 02:46:37.345786 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:37.345790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:37.346913 (+ 1123us) server_negotiation.cc:407] Connection header received
0929 02:46:37.349257 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:37.349263 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:37.349341 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:37.349472 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:37.351380 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.352049 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.354615 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.355053 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.355183 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:37.359236 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:37.359247 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:37.359256 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:37.359305 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:37.363276 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:37.364059 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:37.364069 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:37.364076 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:37.364250 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:37.364749 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:37.364758 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:37.364765 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:37.365193 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:37.365364 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:37.366179 (+ 815us) server_negotiation.cc:299] Negotiation successful
0929 02:46:37.366463 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":189,"threads_started":1}
I20260929 02:46:37.367694 6208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:37.345900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39391 (local address 127.0.97.1:33757)
0929 02:46:37.346705 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:37.346770 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:37.349002 (+ 2232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:37.349675 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:37.349683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:37.350364 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:37.351175 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:37.351215 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.352215 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.352223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:37.354416 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:37.354433 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.355297 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.355306 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:37.355640 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:37.356472 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:37.356496 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:37.358990 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:37.363445 (+ 4455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:37.363453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:37.363467 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:37.363911 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:37.364373 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:37.364377 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:37.364379 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:37.364615 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:37.365356 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:37.365362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:37.365712 (+ 350us) client_negotiation.cc:769] Sending connection context
0929 02:46:37.365997 (+ 285us) client_negotiation.cc:240] Negotiation successful
0929 02:46:37.366423 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":285,"threads_started":1}
I20260929 02:46:37.369575 6205 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39391
I20260929 02:46:37.369949 6205 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:37.370844 6205 heartbeater.cc:511] Master 127.0.97.62:39391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:46:37.373586 6016 ts_manager.cc:194] Registered new tserver with Master: 46c7ae7570ee4b1f8d693a80df73a0c9 (127.0.97.1:36331)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:46:37.375419 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:37.400894 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39391
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:37.728036 6211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:37.728461 6211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:37.728679 6211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:37.736781 6211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:37.736889 6211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:37.737000 6211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:37.746634 6211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:37.747913 6211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:37.749361 6211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:37.763993 6216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:37.764034 6219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:37.763949 6217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:37.765074 6211 server_base.cc:1034] running on GCE node
I20260929 02:46:37.765640 6211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:37.766566 6211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:37.767783 6211 hybrid_clock.cc:648] HybridClock initialized: now 1790649997767721 us; error 53 us; skew 500 ppm
I20260929 02:46:37.768054 6211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:37 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:37.774830 6211 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:37.776384 6211 webserver.cc:460] Webserver started at http://127.0.97.2:39237/ using document root <none> and password file <none>
I20260929 02:46:37.777263 6211 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:37.777421 6211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:37.777724 6211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:37.781129 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "97de698f7b764cada12bfff8a276b0ed"
format_stamp: "Formatted at 2026-09-29 02:46:37 on dist-test-slave-6tjh"
server_key: "5d98d3c74a6f1a1fa62ca90cb6c5926e"
server_key_iv: "2e699d31ca09afcf59e2fa9baf8bc486"
server_key_version: "encryptionkey@0"
I20260929 02:46:37.781958 6211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "97de698f7b764cada12bfff8a276b0ed"
format_stamp: "Formatted at 2026-09-29 02:46:37 on dist-test-slave-6tjh"
server_key: "5d98d3c74a6f1a1fa62ca90cb6c5926e"
server_key_iv: "2e699d31ca09afcf59e2fa9baf8bc486"
server_key_version: "encryptionkey@0"
I20260929 02:46:37.787717 6211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:46:37.791584 6226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:37.793334 6211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:37.793534 6211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "97de698f7b764cada12bfff8a276b0ed"
format_stamp: "Formatted at 2026-09-29 02:46:37 on dist-test-slave-6tjh"
server_key: "5d98d3c74a6f1a1fa62ca90cb6c5926e"
server_key_iv: "2e699d31ca09afcf59e2fa9baf8bc486"
server_key_version: "encryptionkey@0"
I20260929 02:46:37.793784 6211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:37.811976 6211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:37.816341 6211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:37.816702 6211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:37.817680 6211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:37.819073 6211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:37.819175 6211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:37.819336 6211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:37.819409 6211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:37.852928 6211 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34327
I20260929 02:46:37.852972 6338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34327 every 8 connection(s)
I20260929 02:46:37.854238 6211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:37.861624 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 6211
I20260929 02:46:37.861860 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:37.862313 388 external_mini_cluster.cc:1428] Setting key 77b2f9ed604530358c0683269cefb844
Sep 29 02:46:37 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:37.875288 6207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:37.856858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54403 (local address 127.0.97.62:39391)
0929 02:46:37.857027 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:37.857031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:37.858270 (+ 1239us) server_negotiation.cc:407] Connection header received
0929 02:46:37.860031 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:37.860035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:37.860102 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:37.860191 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:37.861843 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.862541 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.864090 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.864505 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.864711 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:37.868613 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:37.868623 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:37.868631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:37.868674 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:37.871754 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:37.872870 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:37.872874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:37.872877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:37.872996 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:37.873570 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:37.873574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:37.873576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:37.873836 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:37.873947 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:37.874875 (+ 928us) server_negotiation.cc:299] Negotiation successful
0929 02:46:37.875039 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:46:37.877094 6341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:37.857369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39391 (local address 127.0.97.2:54403)
0929 02:46:37.858104 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:37.858147 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:37.859774 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:37.860472 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:37.860484 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:37.861110 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:37.861660 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:37.861676 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.862702 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.862711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:37.863925 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:37.863936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:37.864809 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:37.864816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:37.865041 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:37.865719 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:37.865747 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:37.868308 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:37.871922 (+ 3614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:37.871937 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:37.871960 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:37.872746 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:37.873114 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:37.873123 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:37.873130 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:37.873400 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:37.873964 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:37.873976 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:37.874468 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 02:46:37.874813 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 02:46:37.875442 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":311,"threads_started":1}
I20260929 02:46:37.878640 6339 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39391
I20260929 02:46:37.879004 6339 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:37.879864 6339 heartbeater.cc:511] Master 127.0.97.62:39391 requested a full tablet report, sending...
I20260929 02:46:37.881726 6016 ts_manager.cc:194] Registered new tserver with Master: 97de698f7b764cada12bfff8a276b0ed (127.0.97.2:34327)
I20260929 02:46:37.882545 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54403
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:37.912407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39391
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:38.242786 6344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:38.243129 6344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:38.243312 6344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:38.250608 6344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:38.250764 6344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:38.250948 6344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:38.260833 6344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:38.262228 6344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:38.263617 6344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:38.277336 6350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:38.277376 6352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:38.277307 6349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:38.278309 6344 server_base.cc:1034] running on GCE node
I20260929 02:46:38.278973 6344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:38.280035 6344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:38.281278 6344 hybrid_clock.cc:648] HybridClock initialized: now 1790649998281216 us; error 61 us; skew 500 ppm
I20260929 02:46:38.281625 6344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.284917 6344 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:38.286504 6344 webserver.cc:460] Webserver started at http://127.0.97.3:33873/ using document root <none> and password file <none>
I20260929 02:46:38.287168 6344 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:38.287271 6344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:38.287498 6344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:38.290781 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "d7485c0137004ed6b0d85a716ed221c5"
format_stamp: "Formatted at 2026-09-29 02:46:38 on dist-test-slave-6tjh"
server_key: "c1c502ff0c4bc2421e59dfc879a29890"
server_key_iv: "d3d874bca66dfdc508023d94f45d769b"
server_key_version: "encryptionkey@0"
I20260929 02:46:38.291693 6344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d7485c0137004ed6b0d85a716ed221c5"
format_stamp: "Formatted at 2026-09-29 02:46:38 on dist-test-slave-6tjh"
server_key: "c1c502ff0c4bc2421e59dfc879a29890"
server_key_iv: "d3d874bca66dfdc508023d94f45d769b"
server_key_version: "encryptionkey@0"
I20260929 02:46:38.296892 6344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:46:38.300403 6359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:38.301857 6344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:38.302037 6344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "d7485c0137004ed6b0d85a716ed221c5"
format_stamp: "Formatted at 2026-09-29 02:46:38 on dist-test-slave-6tjh"
server_key: "c1c502ff0c4bc2421e59dfc879a29890"
server_key_iv: "d3d874bca66dfdc508023d94f45d769b"
server_key_version: "encryptionkey@0"
I20260929 02:46:38.302251 6344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:38.345674 6344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:38.350227 6344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:38.350551 6344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:38.351532 6344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:38.353011 6344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:38.353121 6344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:38.353276 6344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:38.353349 6344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:38.378799 6205 heartbeater.cc:503] Master 127.0.97.62:39391 was elected leader, sending a full tablet report...
I20260929 02:46:38.387377 6344 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37869
I20260929 02:46:38.387454 6471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37869 every 8 connection(s)
I20260929 02:46:38.388691 6344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:38.393850 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 6344
I20260929 02:46:38.394037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:38.394452 388 external_mini_cluster.cc:1428] Setting key ebef28d52661e8683473f5e25388b2ba
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:38.410290 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.391569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60951 (local address 127.0.97.62:39391)
0929 02:46:38.392075 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.392082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.393394 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 02:46:38.395518 (+ 2124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.395524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.395624 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.395756 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.397443 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.398414 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.400094 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.400519 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.400731 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.404373 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:38.404379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:38.404382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:38.404422 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:38.407217 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.407955 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.407962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.407965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.408085 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.408563 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.408569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.408573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.408836 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:38.408987 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:38.409705 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 02:46:38.409911 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":260,"threads_started":1}
I20260929 02:46:38.411368 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.392287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39391 (local address 127.0.97.3:60951)
0929 02:46:38.393217 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:38.393267 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:38.395230 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:38.396007 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:38.396016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:38.396617 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:38.397270 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:38.397288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.398580 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.398586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:38.399942 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:38.399954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.400808 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.400816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:38.401065 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.401734 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:38.401758 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:38.404126 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:38.407368 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:38.407377 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:38.407394 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:38.407833 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:38.408219 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:38.408224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:38.408228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:38.408423 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:38.408974 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:38.408982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:38.409328 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 02:46:38.409616 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 02:46:38.410163 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":446,"threads_started":1}
I20260929 02:46:38.412858 6472 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39391
I20260929 02:46:38.413177 6472 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:38.413949 6472 heartbeater.cc:511] Master 127.0.97.62:39391 requested a full tablet report, sending...
I20260929 02:46:38.415338 6016 ts_manager.cc:194] Registered new tserver with Master: d7485c0137004ed6b0d85a716ed221c5 (127.0.97.3:37869)
I20260929 02:46:38.416123 6016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60951
I20260929 02:46:38.422243 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/test-admin.keytab.
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.449983 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 02:46:38.455181 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:38.452664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.0.97.1:36331)
0929 02:46:38.453315 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.453325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.453372 (+ 47us) server_negotiation.cc:407] Connection header received
0929 02:46:38.453447 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.453463 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.453683 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.453851 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.454761 (+ 910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52402: BlockingRecv error: recv got EOF from 127.0.0.1:52402 (error 108)
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":257,"threads_started":1}
W20260929 02:46:38.455535 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc)
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.457206 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.458536 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:38.474092 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.460315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52414 (local address 127.0.97.1:36331)
0929 02:46:38.460496 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.460502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.460637 (+ 135us) server_negotiation.cc:407] Connection header received
0929 02:46:38.460857 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.460862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.460930 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.461032 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.462018 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.463011 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.464234 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.464891 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.465029 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.468492 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:38.468503 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:38.468517 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:38.468601 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:38.471275 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.471925 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.471934 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.471938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.472061 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.472443 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.472449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.472453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.472872 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:38.473174 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:38.473497 (+ 323us) server_negotiation.cc:299] Negotiation successful
0929 02:46:38.473780 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 02:46:38.480125 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:38.478273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52428 (local address 127.0.97.1:36331)
0929 02:46:38.478486 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.478493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.478777 (+ 284us) server_negotiation.cc:407] Connection header received
0929 02:46:38.478973 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.478978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.479041 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.479138 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.479868 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52428: BlockingRecv error: recv got EOF from 127.0.0.1:52428 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260929 02:46:38.480439 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc' of type FILE
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.482043 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.483264 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:38.497963 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.484529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.97.1:36331)
0929 02:46:38.484768 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.484775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.485017 (+ 242us) server_negotiation.cc:407] Connection header received
0929 02:46:38.485139 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.485143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.485223 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.485309 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.486267 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.487114 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.488350 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.488868 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.488974 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.492661 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:38.492667 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:38.492672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:38.492756 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:38.495176 (+ 2420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.495940 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.495945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.495957 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.496095 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.496473 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.496477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.496479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.496843 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:38.497041 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:38.497477 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 02:46:38.497694 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260929 02:46:38.502650 6480 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:38.500999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.0.97.1:36331)
0929 02:46:38.501226 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.501231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.501407 (+ 176us) server_negotiation.cc:407] Connection header received
0929 02:46:38.501592 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.501597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.501677 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.501787 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.502417 (+ 630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52446: BlockingRecv error: recv got EOF from 127.0.0.1:52446 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 02:46:38.502858 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790649949495923-388-0/krb5kdc/krb5cc)
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.504333 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:38.505715 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:38.520761 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.507172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52456 (local address 127.0.97.1:36331)
0929 02:46:38.507397 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.507401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.507445 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:46:38.507635 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.507639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.507713 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.507810 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.508933 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.509796 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.511002 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.511479 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.511582 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.515119 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:38.515124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:38.515128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:38.515209 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:38.518087 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.518761 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.518766 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.518769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.518941 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.519333 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.519337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.519339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.519604 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:38.519891 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:38.520159 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 02:46:38.520375 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:46:38.522567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 6077
I20260929 02:46:38.534889 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.523321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52472 (local address 127.0.97.1:36331)
0929 02:46:38.523467 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.523472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.523864 (+ 392us) server_negotiation.cc:407] Connection header received
0929 02:46:38.523974 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.523979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.524040 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.524122 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.525063 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.525739 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.527066 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.527479 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.527573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.529816 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:38.529820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:38.529824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:38.529855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:38.532490 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.533195 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.533199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.533202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.533353 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.533665 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.533668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.533670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.533942 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:38.534148 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:38.534441 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 02:46:38.534616 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:46:38.535490 6200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:38.619086 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 6077
I20260929 02:46:38.635114 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 6211
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:46:38.650115 6493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.635972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.0.97.2:34327)
0929 02:46:38.636448 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.636455 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.636479 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:46:38.636536 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.636582 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.636844 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.637008 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.637959 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.638937 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.640117 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.640723 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.640861 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.644712 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:38.644723 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:38.644738 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:38.644770 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:38.647472 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.648180 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.648187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.648190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.648301 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.648731 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.648736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.648738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.649085 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:38.649316 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:38.649611 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 02:46:38.649835 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":238,"threads_started":1}
I20260929 02:46:38.651362 6334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:38.736070 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 6211
I20260929 02:46:38.751897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 6344
Sep 29 02:46:38 dist-test-slave-6tjh krb5kdc[5971](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790649998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:46:38.770284 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:38.753032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.0.97.3:37869)
0929 02:46:38.753669 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:38.753675 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:38.753704 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:46:38.753763 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:38.753769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:38.753986 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:38.754114 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:38.755071 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.756045 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.757369 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:38.757953 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:38.758053 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:38.762540 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:38.762559 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:38.762580 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:38.762629 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:38.766765 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.767722 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.767731 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.767735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.767882 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:38.768340 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:38.768345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:38.768349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:38.768844 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:38.769176 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:38.769574 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 02:46:38.769854 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":334,"threads_started":1}
I20260929 02:46:38.771684 6467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:38.857583 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 6344
I20260929 02:46:38.873071 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 5984
I20260929 02:46:38.874272 6046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:38.976334 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 5984
2026-09-29T02:46:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3989 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:39 dist-test-slave-6tjh krb5kdc[6500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:39 dist-test-slave-6tjh krb5kdc[6500](info): set up 2 sockets
Sep 29 02:46:39 dist-test-slave-6tjh krb5kdc[6500](info): commencing operation
krb5kdc: starting...
W20260929 02:46:40.097726 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:40 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 02:46:40.314435 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39317
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:40.638226 6513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:40.638512 6513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:40.638600 6513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:40.645823 6513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:40.645939 6513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:40.645985 6513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:40.646036 6513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:40.646081 6513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:40.655795 6513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:40.657177 6513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:40.658421 6513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:40.670305 6518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:40.670266 6519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:40.670225 6521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:40.671229 6513 server_base.cc:1034] running on GCE node
I20260929 02:46:40.672052 6513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:40.673574 6513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:40.674786 6513 hybrid_clock.cc:648] HybridClock initialized: now 1790650000674726 us; error 61 us; skew 500 ppm
I20260929 02:46:40.675176 6513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:40 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:40.678339 6513 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 02:46:40.679873 6513 webserver.cc:460] Webserver started at http://127.0.97.62:34251/ using document root <none> and password file <none>
I20260929 02:46:40.680630 6513 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:40.680778 6513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:40.681082 6513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:40.684012 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "61a426582b774c498b111e9c0b3c2df2"
format_stamp: "Formatted at 2026-09-29 02:46:40 on dist-test-slave-6tjh"
server_key: "2b957b92322662cfc2fc2c8369ba455d"
server_key_iv: "eb5df83500ca3b8e12f90e77482d5995"
server_key_version: "encryptionkey@0"
I20260929 02:46:40.684839 6513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "61a426582b774c498b111e9c0b3c2df2"
format_stamp: "Formatted at 2026-09-29 02:46:40 on dist-test-slave-6tjh"
server_key: "2b957b92322662cfc2fc2c8369ba455d"
server_key_iv: "eb5df83500ca3b8e12f90e77482d5995"
server_key_version: "encryptionkey@0"
I20260929 02:46:40.689837 6513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 02:46:40.693238 6528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:40.694599 6513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:40.694775 6513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "61a426582b774c498b111e9c0b3c2df2"
format_stamp: "Formatted at 2026-09-29 02:46:40 on dist-test-slave-6tjh"
server_key: "2b957b92322662cfc2fc2c8369ba455d"
server_key_iv: "eb5df83500ca3b8e12f90e77482d5995"
server_key_version: "encryptionkey@0"
I20260929 02:46:40.694965 6513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:40.711282 6513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:40.715787 6513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:40.716115 6513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:40.735317 6513 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39317
I20260929 02:46:40.735327 6579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39317 every 8 connection(s)
I20260929 02:46:40.736760 6513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:40.740669 6580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:40.745774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 6513
I20260929 02:46:40.746006 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:40.746656 388 external_mini_cluster.cc:1428] Setting key 01bf51b8180c48e5e8d606a943906f77
I20260929 02:46:40.752172 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: Bootstrap starting.
I20260929 02:46:40.756116 6580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:40.757513 6580 log.cc:824] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:40.761421 6580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: No bootstrap required, opened a new log
Sep 29 02:46:40 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:40.769953 6580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } }
I20260929 02:46:40.770341 6580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:40.770452 6580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61a426582b774c498b111e9c0b3c2df2, State: Initialized, Role: FOLLOWER
I20260929 02:46:40.771147 6580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } }
I20260929 02:46:40.771414 6580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:40.771559 6580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:40.771747 6580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:40.772708 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:40.748789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.97.62:39317)
0929 02:46:40.749732 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:40.749744 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:40.749793 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:46:40.751453 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:40.751477 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:40.751887 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:40.752235 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:40.754265 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:40.755581 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:40.757696 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:40.758785 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:40.759016 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:40.764670 (+ 5654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:40.764680 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:40.764702 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:40.764746 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:40.767910 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:40.768667 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:40.768674 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:40.768684 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:40.768868 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:40.769270 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:40.769274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:40.769276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:40.769841 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:40.770156 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:40.770707 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 02:46:40.771169 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":411,"threads_started":1}
I20260929 02:46:40.774669 6580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } }
I20260929 02:46:40.775228 6580 leader_election.cc:302] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61a426582b774c498b111e9c0b3c2df2; no voters:
I20260929 02:46:40.775970 6580 leader_election.cc:288] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:40.776075 6585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:40.776832 6585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 1 LEADER]: Becoming Leader. State: Replica: 61a426582b774c498b111e9c0b3c2df2, State: Running, Role: LEADER
I20260929 02:46:40.777369 6585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } }
I20260929 02:46:40.778259 6580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:40.782687 6586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } }
I20260929 02:46:40.782828 6587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a426582b774c498b111e9c0b3c2df2. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } }
I20260929 02:46:40.783141 6586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:40.783206 6587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:40.784653 6596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:40.789480 6596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:40.798205 6596 catalog_manager.cc:1347] Generated new cluster ID: c1a2063bb0094966bcca6788eea1c355
I20260929 02:46:40.798352 6596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:40.808660 6596 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:40.810534 6596 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:40.819778 6596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: Generated new TSK 0
I20260929 02:46:40.820747 6596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 02:46:40.881319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39317
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:41.214212 6606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:41.214553 6606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:41.214733 6606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:41.222211 6606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:41.222362 6606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:41.222536 6606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:41.232982 6606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:41.234300 6606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:41.235627 6606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:41.248644 6612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:41.248731 6614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:41.249174 6611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:41.249450 6606 server_base.cc:1034] running on GCE node
I20260929 02:46:41.250063 6606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:41.250985 6606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:41.252215 6606 hybrid_clock.cc:648] HybridClock initialized: now 1790650001252173 us; error 36 us; skew 500 ppm
I20260929 02:46:41.252530 6606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:41 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:41.255749 6606 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 02:46:41.257301 6606 webserver.cc:460] Webserver started at http://127.0.97.1:35571/ using document root <none> and password file <none>
I20260929 02:46:41.258002 6606 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:41.258152 6606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:41.258456 6606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:41.261436 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "ac8ff7c891f041c2b808157ebada8a5c"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f9ee871cd13e33ed64d751fdc91236fd"
server_key_iv: "eb7349fc103586aa8595badc0cd33798"
server_key_version: "encryptionkey@0"
I20260929 02:46:41.262241 6606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ac8ff7c891f041c2b808157ebada8a5c"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f9ee871cd13e33ed64d751fdc91236fd"
server_key_iv: "eb7349fc103586aa8595badc0cd33798"
server_key_version: "encryptionkey@0"
I20260929 02:46:41.267486 6606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:46:41.271402 6621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:41.272939 6606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:41.273146 6606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "ac8ff7c891f041c2b808157ebada8a5c"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f9ee871cd13e33ed64d751fdc91236fd"
server_key_iv: "eb7349fc103586aa8595badc0cd33798"
server_key_version: "encryptionkey@0"
I20260929 02:46:41.273416 6606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:41.285780 6606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:41.290045 6606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:41.290339 6606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:41.291275 6606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:41.292652 6606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:41.292780 6606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:41.292925 6606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:41.293010 6606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:41.326638 6606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40727
I20260929 02:46:41.326726 6733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40727 every 8 connection(s)
I20260929 02:46:41.327908 6606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:41.334035 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 6606
I20260929 02:46:41.334264 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:41.335078 388 external_mini_cluster.cc:1428] Setting key d3c4ad36fb1419c74efd7bd7e3381cd7
Sep 29 02:46:41 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:41.351434 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:41.330859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42639 (local address 127.0.97.62:39317)
0929 02:46:41.331269 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:41.331274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:41.332253 (+ 979us) server_negotiation.cc:407] Connection header received
0929 02:46:41.334219 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:41.334225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:41.334319 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:41.334448 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:41.336456 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.337406 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.340009 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.340413 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.340510 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:41.345159 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:41.345164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:41.345168 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:41.345200 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:41.347902 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:41.348943 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:41.348947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:41.348950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:41.349064 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:41.349562 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:41.349565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:41.349567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:41.349811 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:41.349915 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:41.350917 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0929 02:46:41.351119 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260929 02:46:41.353044 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:41.331399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.1:42639)
0929 02:46:41.332087 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:41.332125 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:41.333944 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:41.334812 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:41.334828 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:41.335566 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:41.336275 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:41.336309 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.337569 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.337574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:41.339835 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:41.339858 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.340662 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.340672 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:41.341011 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:41.341780 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:41.341816 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:41.344883 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:41.348067 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:41.348080 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:41.348103 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:41.348797 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:41.349171 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:41.349178 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:41.349185 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:41.349447 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:41.349940 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:41.349951 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:41.350437 (+ 486us) client_negotiation.cc:769] Sending connection context
0929 02:46:41.350761 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 02:46:41.351391 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260929 02:46:41.354904 6734 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39317
I20260929 02:46:41.355310 6734 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:41.356142 6734 heartbeater.cc:511] Master 127.0.97.62:39317 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:46:41.358765 6545 ts_manager.cc:194] Registered new tserver with Master: ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260929 02:46:41.360399 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 02:46:41.388233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39317
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:41.714998 6740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:41.715349 6740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:41.715514 6740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:41.723032 6740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:41.723217 6740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:41.723471 6740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:41.736719 6740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:41.738035 6740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:41.739305 6740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:41.752727 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:41.752750 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:41.753019 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:41.753427 6740 server_base.cc:1034] running on GCE node
I20260929 02:46:41.754122 6740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:41.755014 6740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:41.756253 6740 hybrid_clock.cc:648] HybridClock initialized: now 1790650001756195 us; error 53 us; skew 500 ppm
I20260929 02:46:41.756722 6740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:41 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:41.759877 6740 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 02:46:41.761374 6740 webserver.cc:460] Webserver started at http://127.0.97.2:37257/ using document root <none> and password file <none>
I20260929 02:46:41.762074 6740 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:41.762233 6740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:41.762529 6740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:41.765513 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "5036ef3454bc434ca0b479d99bb383cb"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f04d71c5b09f004ae71f59a71a9f6641"
server_key_iv: "b71d64b7aa87c4d3731b736abbdae47b"
server_key_version: "encryptionkey@0"
I20260929 02:46:41.766291 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5036ef3454bc434ca0b479d99bb383cb"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f04d71c5b09f004ae71f59a71a9f6641"
server_key_iv: "b71d64b7aa87c4d3731b736abbdae47b"
server_key_version: "encryptionkey@0"
I20260929 02:46:41.771353 6740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:46:41.775125 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:41.776647 6740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:46:41.776863 6740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "5036ef3454bc434ca0b479d99bb383cb"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f04d71c5b09f004ae71f59a71a9f6641"
server_key_iv: "b71d64b7aa87c4d3731b736abbdae47b"
server_key_version: "encryptionkey@0"
I20260929 02:46:41.777092 6740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:41.793543 6740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:41.797911 6740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:41.798228 6740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:41.799331 6740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:41.800724 6740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:41.800850 6740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:41.801005 6740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:41.801077 6740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:41.835184 6740 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42933
I20260929 02:46:41.835223 6867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42933 every 8 connection(s)
I20260929 02:46:41.836529 6740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:41.839671 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 6740
I20260929 02:46:41.840009 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:41.840471 388 external_mini_cluster.cc:1428] Setting key da675bef9ab52a60cd35738d30b54c6b
Sep 29 02:46:41 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650001, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:41.859170 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:41.839160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48681 (local address 127.0.97.62:39317)
0929 02:46:41.839413 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:41.839422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:41.840717 (+ 1295us) server_negotiation.cc:407] Connection header received
0929 02:46:41.843211 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:41.843218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:41.843320 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:41.843453 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:41.845238 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.846507 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.848155 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.849052 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.849234 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:41.853419 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:41.853424 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:41.853427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:41.853460 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:41.856022 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:41.856867 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:41.856879 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:41.856882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:41.857011 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:41.857464 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:41.857468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:41.857470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:41.857725 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:41.857891 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:41.858595 (+ 704us) server_negotiation.cc:299] Negotiation successful
0929 02:46:41.858831 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:46:41.860194 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:41.839664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:48681)
0929 02:46:41.840484 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:41.840566 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:41.842874 (+ 2308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:41.843745 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:41.843759 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:41.844343 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:41.845020 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:41.845039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.846679 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.846683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:41.847984 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:41.847996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:41.849317 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:41.849326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:41.849701 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:41.850545 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:41.850574 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:41.853159 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:41.856176 (+ 3017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:41.856186 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:41.856205 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:41.856740 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:41.857126 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:41.857131 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:41.857135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:41.857336 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:41.857859 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:41.857868 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:41.858231 (+ 363us) client_negotiation.cc:769] Sending connection context
0929 02:46:41.858518 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 02:46:41.859061 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":349,"threads_started":1}
I20260929 02:46:41.861779 6868 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39317
I20260929 02:46:41.862175 6868 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:46:41.863194 6868 heartbeater.cc:511] Master 127.0.97.62:39317 requested a full tablet report, sending...
I20260929 02:46:41.864840 6545 ts_manager.cc:194] Registered new tserver with Master: 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933)
I20260929 02:46:41.865597 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 02:46:41.893153 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39317
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:42.221113 6873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:42.221402 6873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:42.221496 6873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:42.228906 6873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:42.229030 6873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:42.229144 6873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:42.238881 6873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:42.240128 6873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:42.241443 6873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:42.254519 6879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:42.254587 6878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:42.254796 6881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:42.255471 6873 server_base.cc:1034] running on GCE node
I20260929 02:46:42.256094 6873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:42.257215 6873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:42.258476 6873 hybrid_clock.cc:648] HybridClock initialized: now 1790650002258408 us; error 46 us; skew 500 ppm
I20260929 02:46:42.258807 6873 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:42 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:42.262217 6873 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 02:46:42.263700 6873 webserver.cc:460] Webserver started at http://127.0.97.3:36153/ using document root <none> and password file <none>
I20260929 02:46:42.264400 6873 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:42.264591 6873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:42.264914 6873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:42.267992 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "b490e73593544d208b83ca28a1c91135"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "a676e22965b0bdead07f87ded6b9e082"
server_key_iv: "38dae8746fcefb585fa4872a855b0c8a"
server_key_version: "encryptionkey@0"
I20260929 02:46:42.268862 6873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b490e73593544d208b83ca28a1c91135"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "a676e22965b0bdead07f87ded6b9e082"
server_key_iv: "38dae8746fcefb585fa4872a855b0c8a"
server_key_version: "encryptionkey@0"
I20260929 02:46:42.274257 6873 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:42.278041 6888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:42.279621 6873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:42.279805 6873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "b490e73593544d208b83ca28a1c91135"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "a676e22965b0bdead07f87ded6b9e082"
server_key_iv: "38dae8746fcefb585fa4872a855b0c8a"
server_key_version: "encryptionkey@0"
I20260929 02:46:42.280026 6873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:42.306207 6873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:42.310705 6873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:42.311005 6873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:42.311982 6873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:42.313359 6873 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:42.313459 6873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:42.313613 6873 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:42.313686 6873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:42.346349 6873 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46157
I20260929 02:46:42.346395 7000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46157 every 8 connection(s)
I20260929 02:46:42.347574 6873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:42.353791 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 6873
I20260929 02:46:42.353976 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:42.354421 388 external_mini_cluster.cc:1428] Setting key 8c5cc8034f9a97c0fa55adf4fc93caa8
Sep 29 02:46:42 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:42.363926 6734 heartbeater.cc:503] Master 127.0.97.62:39317 was elected leader, sending a full tablet report...
I20260929 02:46:42.369699 6736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.350243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40377 (local address 127.0.97.62:39317)
0929 02:46:42.350416 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:42.350421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:42.351501 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 02:46:42.353388 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:42.353393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:42.353485 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:42.353592 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:42.355314 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.356400 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.358113 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.358513 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.358661 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.362650 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:42.362655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:42.362658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:42.362890 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:42.366146 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.366957 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.366971 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.366973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.367101 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.367567 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.367578 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.367582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.367888 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:42.368063 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:42.369036 (+ 973us) server_negotiation.cc:299] Negotiation successful
0929 02:46:42.369283 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:46:42.370812 7003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.350683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.3:40377)
0929 02:46:42.351346 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:42.351382 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:42.353125 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:42.353816 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:42.353824 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:42.354417 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:42.355135 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.355164 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.356595 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.356600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.357955 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.357968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.358729 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.358750 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.358999 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.359869 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:42.359896 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:42.362432 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:42.366302 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.366313 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.366332 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.366816 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.367230 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.367235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.367239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.367436 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.368035 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:42.368044 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:42.368536 (+ 492us) client_negotiation.cc:769] Sending connection context
0929 02:46:42.368918 (+ 382us) client_negotiation.cc:240] Negotiation successful
0929 02:46:42.369549 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":266,"threads_started":1}
I20260929 02:46:42.372455 7001 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39317
I20260929 02:46:42.372813 7001 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:42.373553 7001 heartbeater.cc:511] Master 127.0.97.62:39317 requested a full tablet report, sending...
I20260929 02:46:42.375057 6545 ts_manager.cc:194] Registered new tserver with Master: b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157)
I20260929 02:46:42.375906 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:40377
I20260929 02:46:42.382740 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260929 02:46:42.433542 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35441
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39317,127.0.97.61:35441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:42.755051 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:42.755363 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:42.755523 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:42.762693 7006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:42.762843 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:42.762948 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:42.763041 7006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:42.763123 7006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:42.773373 7006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39317,127.0.97.61:35441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:35441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
W20260929 02:46:42.773778 7006 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:39317,127.0.97.61:35441'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 02:46:42.774827 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:42.776142 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:42.788220 7011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:42.788189 7014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:42.788275 7012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:42.789417 7006 server_base.cc:1034] running on GCE node
I20260929 02:46:42.790095 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:42.791471 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:42.792660 7006 hybrid_clock.cc:648] HybridClock initialized: now 1790650002792575 us; error 56 us; skew 500 ppm
I20260929 02:46:42.793032 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:42 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:42.800344 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 02:46:42.802030 7006 webserver.cc:460] Webserver started at http://127.0.97.61:37131/ using document root <none> and password file <none>
I20260929 02:46:42.802773 7006 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:42.802866 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:42.803200 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:42.806218 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/instance:
uuid: "c22ecfc2606d4348b278ef212d1d20c8"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "0752a3cd2791d9781b38c94540740f94"
server_key_iv: "0660feecf76f70a0488fce889b279c9d"
server_key_version: "encryptionkey@0"
I20260929 02:46:42.806979 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal/instance:
uuid: "c22ecfc2606d4348b278ef212d1d20c8"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "0752a3cd2791d9781b38c94540740f94"
server_key_iv: "0660feecf76f70a0488fce889b279c9d"
server_key_version: "encryptionkey@0"
I20260929 02:46:42.812439 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:46:42.816150 7021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:42.817629 7006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:46:42.817833 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
uuid: "c22ecfc2606d4348b278ef212d1d20c8"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "0752a3cd2791d9781b38c94540740f94"
server_key_iv: "0660feecf76f70a0488fce889b279c9d"
server_key_version: "encryptionkey@0"
I20260929 02:46:42.818064 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:42.836844 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:42.841015 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:42.841281 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:42.859905 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35441
I20260929 02:46:42.859929 7072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35441 every 8 connection(s)
I20260929 02:46:42.861285 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
I20260929 02:46:42.863694 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 7006
I20260929 02:46:42.865429 7073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:42.868439 6868 heartbeater.cc:503] Master 127.0.97.62:39317 was elected leader, sending a full tablet report...
I20260929 02:46:42.873777 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:46:42 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 02:46:42 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:42.887991 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.865971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.97.61:35441)
0929 02:46:42.866688 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:42.866700 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:42.866744 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:46:42.868338 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:42.868365 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:42.868881 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:42.869320 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:42.870866 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.872357 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.873656 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.874347 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.874545 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.879856 (+ 5311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:42.879867 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:42.879881 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:42.879925 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:42.883721 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.884493 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.884502 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.884512 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.884745 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.885160 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.885167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.885173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.885805 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:42.886068 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:42.886552 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 02:46:42.886978 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":325,"threads_started":1}
I20260929 02:46:42.892406 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.876348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44854 (local address 127.0.97.62:39317)
0929 02:46:42.876960 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:42.876967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:42.877427 (+ 460us) server_negotiation.cc:407] Connection header received
0929 02:46:42.877627 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:42.877632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:42.877693 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:42.877813 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:42.879242 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.880404 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.882333 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.882971 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.883111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.887009 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:42.887014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:42.887017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:42.887050 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:42.889749 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.890482 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.890486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.890489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.890599 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.890967 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.890971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.890975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.891254 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:42.891406 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:42.891833 (+ 427us) server_negotiation.cc:299] Negotiation successful
0929 02:46:42.892052 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":293,"threads_started":1}
I20260929 02:46:42.892450 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.876428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:44854)
0929 02:46:42.877298 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:42.877336 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:42.877503 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:42.877874 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:42.877881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:42.878752 (+ 871us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:42.879076 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.879091 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.880591 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.880598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.882187 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.882208 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.883191 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.883199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.883437 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.884039 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:42.884056 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:42.886799 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:42.889896 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.889909 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.889916 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.890348 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.890709 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.890715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.890718 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.890847 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.891379 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:42.891386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:42.891555 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 02:46:42.891734 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 02:46:42.892047 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":485,"threads_started":1}
I20260929 02:46:42.894130 7073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:46:42 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650002, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 02:46:42.905624 7078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.894722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.0.1:50474)
0929 02:46:42.894909 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:42.894924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:42.895009 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:42.895343 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:42.895346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:42.895649 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:42.895776 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.895783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.896760 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.896763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.897627 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.897637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.898277 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.898282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.898408 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.898936 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:42.898952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:42.901002 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:42.903575 (+ 2573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.903579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.903581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.903926 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.904241 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.904244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.904246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.904361 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.904914 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:42.904917 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:42.905045 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 02:46:42.905173 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 02:46:42.905393 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:46:42.905907 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.894961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.0.97.61:35441)
0929 02:46:42.895098 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:42.895102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:42.895115 (+ 13us) server_negotiation.cc:407] Connection header received
0929 02:46:42.895151 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:42.895154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:42.895203 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:42.895279 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:42.895889 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.896640 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.897768 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.898136 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.898222 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.901150 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:42.901155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:42.901158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:42.901183 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:42.903456 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.904020 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.904024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.904026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.904146 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.904448 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.904451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.904453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.904806 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:42.904909 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:42.905245 (+ 336us) server_negotiation.cc:299] Negotiation successful
0929 02:46:42.905519 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":49}
I20260929 02:46:42.910481 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Bootstrap starting.
I20260929 02:46:42.913508 7073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:42.914444 7073 log.cc:824] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:42.917035 7073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: No bootstrap required, opened a new log
I20260929 02:46:42.924073 7073 raft_consensus.cc:348] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } }
I20260929 02:46:42.924386 7073 raft_consensus.cc:374] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:42.924467 7073 raft_consensus.cc:729] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22ecfc2606d4348b278ef212d1d20c8, State: Initialized, Role: FOLLOWER
I20260929 02:46:42.925055 7073 consensus_queue.cc:260] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } }
I20260929 02:46:42.925895 7080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } } }
I20260929 02:46:42.926209 7080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:46:42.926858 7073 sys_catalog.cc:564] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:46:42.935791 7091 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:46:42.935935 7091 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:46:42.937561 7006 master_runner.cc:428] Detected that this master c22ecfc2606d4348b278ef212d1d20c8 is joining an existing cluster
I20260929 02:46:42.937673 7006 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:35441
I20260929 02:46:42.958669 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.941290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44864 (local address 127.0.97.62:39317)
0929 02:46:42.941511 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:42.941519 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:42.942455 (+ 936us) server_negotiation.cc:407] Connection header received
0929 02:46:42.942723 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:42.942729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:42.942814 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:42.942882 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:42.944467 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.945664 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.947841 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.948613 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.948709 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.951550 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:42.951559 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:42.951566 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:42.951609 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:42.955288 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.956104 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.956112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.956118 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.956273 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.956746 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.956754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.956760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.957130 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:42.957267 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:42.957948 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 02:46:42.958207 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:46:42.959959 7097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.941277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:44864)
0929 02:46:42.942351 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:42.942374 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:42.942564 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:42.943100 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:42.943106 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:42.943859 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:42.944296 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.944312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.945835 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.945843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.947681 (+ 1838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.947697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.948982 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.948990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.949212 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.949926 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:42.949953 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:42.951337 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:42.955440 (+ 4103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.955448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.955455 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.955972 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.956415 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.956422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.956429 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.956620 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.957369 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:42.957377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:42.957618 (+ 241us) client_negotiation.cc:769] Sending connection context
0929 02:46:42.958968 (+ 1350us) client_negotiation.cc:240] Negotiation successful
0929 02:46:42.959348 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"thread_start_us":730,"threads_started":1}
I20260929 02:46:42.965461 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.942652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.0.97.61:35441)
0929 02:46:42.948711 (+ 6059us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:42.948717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:42.949397 (+ 680us) server_negotiation.cc:407] Connection header received
0929 02:46:42.950024 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:42.950030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:42.950096 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:42.950184 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:42.951626 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.953016 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.955221 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.955921 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.956025 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.958369 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:42.958375 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:42.958380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:42.958412 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:42.962073 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.962878 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.962882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.962886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.963027 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.963725 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.963731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.963734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.964257 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:42.964406 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:42.964902 (+ 496us) server_negotiation.cc:299] Negotiation successful
0929 02:46:42.965112 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5927}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 02:46:42.966392 7098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.942778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.0.1:50478)
0929 02:46:42.949293 (+ 6515us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:42.949312 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:42.949455 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:42.950358 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:42.950367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:42.951169 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:42.951489 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.951500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.953218 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.953223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.955049 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.955064 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.956308 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.956314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.956491 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.957109 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:42.957131 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:42.958186 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:42.962240 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.962244 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.962248 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.962756 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.963178 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.963183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.963186 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.963602 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.964487 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:42.964491 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:42.964644 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 02:46:42.965668 (+ 1024us) client_negotiation.cc:240] Negotiation successful
0929 02:46:42.965950 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6388,"thread_start_us":7040,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 29 02:46:42 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 02:46:42.982699 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.966907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.0.1:50482)
0929 02:46:42.967524 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:42.967561 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:42.967701 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:42.968371 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:42.968376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:42.969135 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:42.969352 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.969362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.970588 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.970595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.971587 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:42.971598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.972517 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.972523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:42.972768 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.973395 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:42.973412 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:42.975453 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:42.979300 (+ 3847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.979315 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.979326 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.979933 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.980339 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:42.980347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:42.980354 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:42.980522 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:42.981361 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:42.981370 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:42.981588 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 02:46:42.981846 (+ 258us) client_negotiation.cc:240] Negotiation successful
0929 02:46:42.982222 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":413,"threads_started":1}
I20260929 02:46:42.983709 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:42.967290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50482 (local address 127.0.97.61:35441)
0929 02:46:42.968040 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:42.968045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:42.968062 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:46:42.968135 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:42.968139 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:42.968232 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:42.968333 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:42.969522 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.970462 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.971725 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:42.972340 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:42.972495 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:42.975630 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:42.975635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:42.975638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:42.975665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:42.979159 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.980058 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.980064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.980067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.980219 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:42.980673 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:42.980681 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:42.980687 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:42.981133 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:42.981369 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:42.983093 (+ 1724us) server_negotiation.cc:299] Negotiation successful
0929 02:46:42.983311 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523}
I20260929 02:46:42.986017 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:35441: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 35441
}
attrs {
promote: true
}
}
dest_uuid: "61a426582b774c498b111e9c0b3c2df2"
cas_config_opid_index: -1
I20260929 02:46:42.987214 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } }
I20260929 02:46:42.996335 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER c22ecfc2606d4348b278ef212d1d20c8 (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } }
I20260929 02:46:42.997578 7048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:43.000873 7103 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:35441
I20260929 02:46:43.001534 7048 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Refusing update from remote peer 61a426582b774c498b111e9c0b3c2df2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 02:46:43.001662 7006 master.cc:537] Master@127.0.97.61:35441 shutting down...
I20260929 02:46:43.002696 7103 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:43.002499 7104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } }
I20260929 02:46:43.003175 7104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
W20260929 02:46:43.006820 6532 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.60.keytab.
I20260929 02:46:43.010121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41649
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:46:43.016848 7006 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260929 02:46:43.017096 6532 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 -> Peer c22ecfc2606d4348b278ef212d1d20c8 (127.0.97.61:35441): Couldn't send request to peer c22ecfc2606d4348b278ef212d1d20c8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:46:43.017315 7006 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:43.017510 7006 tablet_replica.cc:330] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: stopping tablet replica
I20260929 02:46:43.023380 7006 master.cc:552] Master@127.0.97.61:35441 shutdown complete.
I20260929 02:46:43.023527 7006 master_runner.cc:305] Clearing existing system tablet
I20260929 02:46:43.024026 7006 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:43.024192 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:43.027617 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:46:43.032899 7109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:43.033322 7006 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:43.033830 7006 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:46:43.033981 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
uuid: "c22ecfc2606d4348b278ef212d1d20c8"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "0752a3cd2791d9781b38c94540740f94"
server_key_iv: "0660feecf76f70a0488fce889b279c9d"
server_key_version: "encryptionkey@0"
I20260929 02:46:43.039794 7006 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:43.046765 7006 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:46:43.047022 7006 log.cc:1196] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 02:46:43.047531 7006 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Deleting consensus metadata
I20260929 02:46:43.047835 7006 master_runner.cc:315] Copying system tablet from 127.0.97.62:39317
I20260929 02:46:43.050474 7006 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39317
I20260929 02:46:43.068928 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.051367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.97.62:39317)
0929 02:46:43.051547 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.051552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.053027 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 02:46:43.053313 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.053321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.053429 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.053557 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.054611 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.055774 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.057525 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.058235 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.058383 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.061659 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.061666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.061671 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.061714 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.065676 (+ 3962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.066598 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.066607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.066612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.066778 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.067229 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.067236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.067241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.067614 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.067785 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.068239 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.068489 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:46:43.069339 7115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.051307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:44874)
0929 02:46:43.052913 (+ 1606us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.052933 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.053144 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.053595 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.053599 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.054111 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.054420 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.054436 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.055941 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.055946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.057366 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.057380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.058412 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.058419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.058589 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.059409 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.059435 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.061345 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.065858 (+ 4513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.065865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.065869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.066422 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.066912 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.066918 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.066923 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.067096 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.067751 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.067756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.067943 (+ 187us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.068096 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.068448 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1478,"thread_start_us":1383,"threads_started":1}
I20260929 02:46:43.070215 6565 tablet_copy_service.cc:140] P 61a426582b774c498b111e9c0b3c2df2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c22ecfc2606d4348b278ef212d1d20c8 ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44874)
I20260929 02:46:43.070669 6565 tablet_copy_service.cc:161] P 61a426582b774c498b111e9c0b3c2df2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c22ecfc2606d4348b278ef212d1d20c8 at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44874: session id = c22ecfc2606d4348b278ef212d1d20c8-00000000000000000000000000000000
I20260929 02:46:43.074438 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:46:43.077080 7006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:46:43.084786 7006 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: tablet copy: Starting download of 0 data blocks...
I20260929 02:46:43.085182 7006 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: tablet copy: Starting download of 1 WAL segments...
I20260929 02:46:43.088131 7006 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:46:43.093165 6565 tablet_copy_service.cc:342] P 61a426582b774c498b111e9c0b3c2df2: Request end of tablet copy session c22ecfc2606d4348b278ef212d1d20c8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:44874
I20260929 02:46:43.093405 6565 tablet_copy_service.cc:433] P 61a426582b774c498b111e9c0b3c2df2: ending tablet copy session c22ecfc2606d4348b278ef212d1d20c8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c22ecfc2606d4348b278ef212d1d20c8
I20260929 02:46:43.105209 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:43.108081 7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:43.108580 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:43.113310 7006 server_base.cc:1034] running on GCE node
W20260929 02:46:43.113701 7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:43.114230 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:43.114976 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:43.116243 7006 hybrid_clock.cc:648] HybridClock initialized: now 1790650003116185 us; error 34 us; skew 500 ppm
I20260929 02:46:43.116585 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:43.118806 7006 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260929 02:46:43.120026 7006 webserver.cc:460] Webserver started at http://127.0.97.61:34769/ using document root <none> and password file <none>
I20260929 02:46:43.120393 7006 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:43.120477 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:43.123554 7006 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:43.125568 7126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:43.126022 7006 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 02:46:43.126163 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
uuid: "c22ecfc2606d4348b278ef212d1d20c8"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "0752a3cd2791d9781b38c94540740f94"
server_key_iv: "0660feecf76f70a0488fce889b279c9d"
server_key_version: "encryptionkey@0"
I20260929 02:46:43.126328 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:43.165714 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:43.176296 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:43.229143 7006 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:35441
I20260929 02:46:43.229193 7177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:35441 every 8 connection(s)
I20260929 02:46:43.230150 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
I20260929 02:46:43.235733 7178 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:46:43.240600 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Bootstrap starting.
I20260929 02:46:43.257627 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:46:43.258253 7178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Bootstrap complete.
I20260929 02:46:43.259388 7178 raft_consensus.cc:348] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } }
I20260929 02:46:43.259641 7178 raft_consensus.cc:729] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c22ecfc2606d4348b278ef212d1d20c8, State: Initialized, Role: LEARNER
I20260929 02:46:43.259914 7178 consensus_queue.cc:260] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } }
I20260929 02:46:43.260731 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } }
I20260929 02:46:43.260986 7178 sys_catalog.cc:564] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:43.261106 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:46:43.270995 7192 catalog_manager.cc:1259] Loaded cluster ID: c1a2063bb0094966bcca6788eea1c355
I20260929 02:46:43.271119 7192 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: loading cluster ID for follower catalog manager: success
I20260929 02:46:43.274893 7192 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: acquiring CA information for follower catalog manager: success
I20260929 02:46:43.277510 7192 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:46:43.378816 7001 heartbeater.cc:503] Master 127.0.97.62:39317 was elected leader, sending a full tablet report...
W20260929 02:46:43.433779 7107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:43.434146 7107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:43.434299 7107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:43.442072 7107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:43.442224 7107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:43.442335 7107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:43.442440 7107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:43.442520 7107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:43.452469 7107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:43.454368 7107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:43.455775 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:43.468161 7200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:43.468115 7198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:43.468364 7197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:43.469427 7107 server_base.cc:1034] running on GCE node
I20260929 02:46:43.470253 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:43.471688 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:43.472935 7107 hybrid_clock.cc:648] HybridClock initialized: now 1790650003472889 us; error 56 us; skew 500 ppm
I20260929 02:46:43.473347 7107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:43.477285 7107 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 02:46:43.479133 7107 webserver.cc:460] Webserver started at http://127.0.97.60:40155/ using document root <none> and password file <none>
I20260929 02:46:43.479930 7107 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:43.480058 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:43.480408 7107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:43.481417 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.474049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50484 (local address 127.0.97.61:35441)
0929 02:46:43.474652 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.474659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.474678 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:46:43.474751 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.474756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.474840 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.474980 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:43.475458 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.476863 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.479263 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.480366 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.480497 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.480749 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.480994 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.481089 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":290,"threads_started":1}
I20260929 02:46:43.481426 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.473859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.0.1:50484)
0929 02:46:43.474039 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.474054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.474149 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.475013 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.475019 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.475051 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:43.475255 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.475269 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.477008 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.477014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.479081 (+ 2067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.479098 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.480529 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.480535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.480804 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.480870 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.480976 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.481126 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:46:43.483078 7153 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 02:46:43.483704 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a426582b774c498b111e9c0b3c2df2. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } }
I20260929 02:46:43.484108 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:46:43.485213 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/instance:
uuid: "9db8f5e40ebb4a5c9f668362bc612a2e"
format_stamp: "Formatted at 2026-09-29 02:46:43 on dist-test-slave-6tjh"
server_key: "cf9f117d88af17ec5252cd48a46224c8"
server_key_iv: "9cda0721fc26fa49333ca11ac264e60b"
server_key_version: "encryptionkey@0"
I20260929 02:46:43.485997 7107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal/instance:
uuid: "9db8f5e40ebb4a5c9f668362bc612a2e"
format_stamp: "Formatted at 2026-09-29 02:46:43 on dist-test-slave-6tjh"
server_key: "cf9f117d88af17ec5252cd48a46224c8"
server_key_iv: "9cda0721fc26fa49333ca11ac264e60b"
server_key_version: "encryptionkey@0"
I20260929 02:46:43.491137 7107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:46:43.494606 7208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:43.496157 7107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:43.496307 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
uuid: "9db8f5e40ebb4a5c9f668362bc612a2e"
format_stamp: "Formatted at 2026-09-29 02:46:43 on dist-test-slave-6tjh"
server_key: "cf9f117d88af17ec5252cd48a46224c8"
server_key_iv: "9cda0721fc26fa49333ca11ac264e60b"
server_key_version: "encryptionkey@0"
I20260929 02:46:43.496474 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:43.507664 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:43.512328 7107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:43.512660 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:43.533743 7107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41649
I20260929 02:46:43.533739 7259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41649 every 8 connection(s)
I20260929 02:46:43.535318 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
I20260929 02:46:43.536871 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 7107
I20260929 02:46:43.540163 7260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:43.548410 7260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:43.562526 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.539126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.97.60:41649)
0929 02:46:43.540440 (+ 1314us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.540461 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.540587 (+ 126us) server_negotiation.cc:407] Connection header received
0929 02:46:43.542357 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.542378 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.542812 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.543157 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.544992 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.546485 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.548149 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.548952 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.549096 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.554071 (+ 4975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.554078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.554088 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.554127 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.557561 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.558378 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.558384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.558391 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.558550 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.558986 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.558990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.558993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.559584 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.559972 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.560481 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.561149 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1071,"spinlock_wait_cycles":11136,"thread_start_us":357,"threads_started":1}
I20260929 02:46:43.569566 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.551969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.97.62:39317)
0929 02:46:43.552201 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.552210 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.552983 (+ 773us) server_negotiation.cc:407] Connection header received
0929 02:46:43.553267 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.553271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.553334 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.553424 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.554711 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.555401 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.558427 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.559046 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.559198 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.563291 (+ 4093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.563299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.563305 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.563345 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.566316 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.567482 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.567487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.567491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.567623 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.568010 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.568015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.568017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.568283 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.568436 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.568959 (+ 523us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.569180 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:46:43.569617 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.551948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:44878)
0929 02:46:43.552827 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.552890 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.553120 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.553523 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.553531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.554275 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.554560 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.554569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.555539 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.555546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.558280 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.558296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.559231 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.559237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.559422 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.560114 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.560130 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.563081 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.566456 (+ 3375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.566469 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.566476 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.567343 (+ 867us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.567745 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.567750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.567754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.567886 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.568412 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.568419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.568662 (+ 243us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.568852 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.569180 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"spinlock_wait_cycles":26496,"thread_start_us":422,"threads_started":1}
I20260929 02:46:43.570931 7260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 02:46:43.583945 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.571528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.0.1:50496)
0929 02:46:43.571709 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.571725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.571813 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.572080 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.572083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.572399 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.572530 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.572537 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.573580 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.573583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.574491 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.574501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.575271 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.575275 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.575407 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.576020 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.576036 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.578163 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.581539 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.581543 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.581546 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.582071 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.582484 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.582487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.582490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.582618 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.583190 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.583193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.583312 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.583424 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.583644 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:46:43.584249 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.571694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50496 (local address 127.0.97.61:35441)
0929 02:46:43.571851 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.571855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.571870 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:46:43.571916 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.571919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.571973 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.572050 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.572743 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.573455 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.574636 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.575113 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.575231 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.578343 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.578348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.578351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.578381 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.581368 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.582231 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.582237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.582244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.582377 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.582774 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.582778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.582780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.583064 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.583185 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.583571 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.583861 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:46:43.585326 7260 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 02:46:43.598464 7264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.586084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.0.1:44890)
0929 02:46:43.586263 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.586279 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.586393 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.586721 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.586725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.587053 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.587178 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.587186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.588095 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.588103 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.589016 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.589027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.589721 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.589726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.589906 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.590430 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.590450 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.592749 (+ 2299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.596007 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.596014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.596018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.596366 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.596785 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.596791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.596794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.596944 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.597431 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.597435 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.597591 (+ 156us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.597730 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.598001 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:46:43.598502 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.586307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.0.97.60:41649)
0929 02:46:43.586459 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.586465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.586480 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:46:43.586517 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.586521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.586576 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.586673 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.587301 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.587979 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.589143 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.589563 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.589701 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.592936 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.592941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.592945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.592971 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.595860 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.596480 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.596487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.596491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.596679 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.597046 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.597053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.597057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.597314 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.597457 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.597815 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.598141 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":123,"server-negotiator.queue_time_us":51}
I20260929 02:46:43.604187 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Bootstrap starting.
I20260929 02:46:43.607323 7260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:43.608327 7260 log.cc:824] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:43.610984 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: No bootstrap required, opened a new log
I20260929 02:46:43.618531 7260 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } }
I20260929 02:46:43.618857 7260 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:43.618994 7260 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9db8f5e40ebb4a5c9f668362bc612a2e, State: Initialized, Role: FOLLOWER
I20260929 02:46:43.619588 7260 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } }
I20260929 02:46:43.620398 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } } }
I20260929 02:46:43.620762 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:46:43.621454 7260 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:46:43.630430 7277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:46:43.630571 7277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:46:43.634613 7107 master_runner.cc:428] Detected that this master 9db8f5e40ebb4a5c9f668362bc612a2e is joining an existing cluster
I20260929 02:46:43.634771 7107 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:41649
I20260929 02:46:43.658779 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.638967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.0.97.60:41649)
0929 02:46:43.639154 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.639163 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.639975 (+ 812us) server_negotiation.cc:407] Connection header received
0929 02:46:43.640175 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.640187 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.640292 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.640411 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.642236 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.643764 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.646881 (+ 3117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.647755 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.647887 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.651182 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.651191 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.651198 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.651244 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.655575 (+ 4331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.656384 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.656389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.656392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.656594 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.657057 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.657061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.657064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.657483 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.657606 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.658087 (+ 481us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.658326 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":69}
I20260929 02:46:43.658850 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.638974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44888 (local address 127.0.97.62:39317)
0929 02:46:43.639158 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.639164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.640689 (+ 1525us) server_negotiation.cc:407] Connection header received
0929 02:46:43.640947 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.640952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.641040 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.641110 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.642619 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.643755 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.646464 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.647118 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.647221 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.650072 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.650079 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.650084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.650125 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.653829 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.656417 (+ 2588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.656423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.656427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.656617 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.657100 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.657107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.657110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.657459 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.657596 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.658201 (+ 605us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.658470 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:46:43.660142 7283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.639129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.0.1:44902)
0929 02:46:43.639869 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.639894 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.640039 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.640614 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.640618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.641516 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.642033 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.642050 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.644031 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.644039 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.646678 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.646697 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.648291 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.648297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.648587 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.649414 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.649447 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.650961 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.655694 (+ 4733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.655700 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.655704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.656261 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.656751 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.656755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.656761 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.656958 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.657666 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.657670 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.657863 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.659286 (+ 1423us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.659596 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":4,"thread_start_us":454,"threads_started":1}
I20260929 02:46:43.660467 7284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.639228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:44888)
0929 02:46:43.640587 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.640609 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.640789 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.641309 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.641313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.642082 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.642437 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.642450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.643963 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.643968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.646278 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.646294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.647546 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.647557 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.647753 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.648504 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.648531 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.649860 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.654005 (+ 4145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.654010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.654015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.656280 (+ 2265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.656785 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.656790 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.656794 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.656973 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.657683 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.657688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.657922 (+ 234us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.659396 (+ 1474us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.659816 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1221,"mutex_wait_us":185,"thread_start_us":1056,"threads_started":1}
I20260929 02:46:43.669374 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 6606
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 02:46:43.690797 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.670559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54962 (local address 127.0.97.1:40727)
0929 02:46:43.671305 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.671314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.671352 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:46:43.671429 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.671438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.671755 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.672002 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.673865 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.675221 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.677537 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.678459 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.678627 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.683959 (+ 5332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.683973 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.683988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.684027 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.687755 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.688479 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.688489 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.688492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.688690 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.689101 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.689106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.689109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.689482 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.689790 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.690121 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.690430 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":387,"threads_started":1}
I20260929 02:46:43.692142 6729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:43.696096 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.640791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.0.1:50506)
0929 02:46:43.654576 (+ 13785us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.654597 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.654739 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.655162 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.655167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.656396 (+ 1229us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.656851 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.656869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.658268 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.658273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.659941 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.659954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.662672 (+ 2718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.662678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.662851 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.663762 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.663779 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.665054 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.669984 (+ 4930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.669990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.669995 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.693236 (+ 23241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.693838 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.693845 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.693849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.694044 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.694821 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.694826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.695074 (+ 248us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.695239 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.695575 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13626,"thread_start_us":13489,"threads_started":1}
I20260929 02:46:43.697146 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.640789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50506 (local address 127.0.97.61:35441)
0929 02:46:43.640961 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.640966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.654715 (+ 13749us) server_negotiation.cc:407] Connection header received
0929 02:46:43.654867 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.654875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.654963 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.655072 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.657033 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.658097 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.660116 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.660818 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.660917 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.665320 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.665329 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.665335 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.665380 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.669834 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.693468 (+ 23634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.693477 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.693484 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.693673 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.694210 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.694215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.694219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.694587 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.694753 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.696528 (+ 1775us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.696824 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:46:43.699044 7100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.666605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.0.1:44914)
0929 02:46:43.666899 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.666918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.667038 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.667556 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.667561 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.668033 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.668226 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.668238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.671865 (+ 3627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.671871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.673595 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.673615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.674765 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.674773 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.674970 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.675706 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.675737 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.678944 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.696337 (+ 17393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.696343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.696346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.696829 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.697186 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.697194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.697199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.697363 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.698001 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.698006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.698172 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.698327 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.698613 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260929 02:46:43.699157 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.666661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.0.97.60:41649)
0929 02:46:43.666904 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.666913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.666972 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:46:43.667248 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.667256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.667394 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.667510 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.670622 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.671691 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.673832 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.674548 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.674691 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.679198 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.679207 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.679217 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.679275 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.696108 (+ 16833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.696967 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.696970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.696972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.697068 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.697494 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.697502 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.697505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.697879 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.698038 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.698528 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.698753 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 02:46:43.700719 6545 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:41649: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 41649
}
attrs {
promote: true
}
}
dest_uuid: "61a426582b774c498b111e9c0b3c2df2"
cas_config_opid_index: 5
I20260929 02:46:43.701809 6545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } }
I20260929 02:46:43.703589 7103 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.705250 7153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 LEARNER]: Refusing update from remote peer 61a426582b774c498b111e9c0b3c2df2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:46:43.705767 7235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:43.707876 7103 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:41649
I20260929 02:46:43.708602 7107 master.cc:537] Master@127.0.97.60:41649 shutting down...
I20260929 02:46:43.710417 7288 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 02:46:43.710966 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.711300 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:43.723074 7153 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.730150 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: true } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.730465 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:46:43.730454 7235 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Refusing update from remote peer 61a426582b774c498b111e9c0b3c2df2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:46:43.731559 7289 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: attempting to promote NON_VOTER c22ecfc2606d4348b278ef212d1d20c8 to VOTER
I20260929 02:46:43.732443 7289 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } }
I20260929 02:46:43.735164 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 LEARNER]: Refusing update from remote peer 61a426582b774c498b111e9c0b3c2df2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 02:46:43.735802 7289 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 02:46:43.739885 7289 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c22ecfc2606d4348b278ef212d1d20c8 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.742125 7107 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:43.742638 7107 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:43.742878 7107 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: stopping tablet replica
I20260929 02:46:43.750933 7289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.751246 7289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:43.759936 7107 master.cc:552] Master@127.0.97.60:41649 shutdown complete.
I20260929 02:46:43.760025 7107 master_runner.cc:305] Clearing existing system tablet
I20260929 02:46:43.760421 7107 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:43.760612 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260929 02:46:43.768208 6531 consensus_peers.cc:487] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 -> Peer 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60:41649): Couldn't send request to peer 9db8f5e40ebb4a5c9f668362bc612a2e. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:46:43.776851 7107 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.000s sys 0.004s
I20260929 02:46:43.777979 7103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.778294 7103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:43.778975 7295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:43.778752 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c22ecfc2606d4348b278ef212d1d20c8 (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.779273 7107 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:43.779510 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 02:46:43.779623 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
uuid: "9db8f5e40ebb4a5c9f668362bc612a2e"
format_stamp: "Formatted at 2026-09-29 02:46:43 on dist-test-slave-6tjh"
server_key: "cf9f117d88af17ec5252cd48a46224c8"
server_key_iv: "9cda0721fc26fa49333ca11ac264e60b"
server_key_version: "encryptionkey@0"
I20260929 02:46:43.784777 7181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.784967 7107 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:43.785072 7181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:46:43.789767 7107 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:46:43.789917 7107 log.cc:1196] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 02:46:43.790400 7107 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Deleting consensus metadata
I20260929 02:46:43.790673 7107 master_runner.cc:315] Copying system tablet from 127.0.97.62:39317
I20260929 02:46:43.794253 7107 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:39317
I20260929 02:46:43.811390 7077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.795186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.97.62:39317)
0929 02:46:43.795441 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:43.795449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:43.795947 (+ 498us) server_negotiation.cc:407] Connection header received
0929 02:46:43.796162 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:43.796168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:43.796246 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:43.796338 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:43.797610 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.798779 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.801472 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.802133 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.802264 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.804765 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:43.804772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:43.804777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:43.804815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:43.808308 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.809198 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.809204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.809208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.809373 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:43.809840 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:43.809845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:43.809849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:43.810179 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:43.810317 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:43.810779 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 02:46:43.811027 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260929 02:46:43.811717 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:43.795095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:44904)
0929 02:46:43.795813 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:43.795839 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:43.796003 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:43.796396 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:43.796402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:43.797140 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:43.797421 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.797435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.798995 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.799003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.801296 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:43.801314 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:43.802311 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:43.802318 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:43.802503 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:43.803305 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:43.803332 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:43.804459 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:43.808471 (+ 4012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.808477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.808482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.809056 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.809505 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:43.809510 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:43.809515 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:43.809716 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:43.810307 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:43.810313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:43.810502 (+ 189us) client_negotiation.cc:769] Sending connection context
0929 02:46:43.810717 (+ 215us) client_negotiation.cc:240] Negotiation successful
0929 02:46:43.811089 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":470,"threads_started":1}
I20260929 02:46:43.812736 6565 tablet_copy_service.cc:140] P 61a426582b774c498b111e9c0b3c2df2: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9db8f5e40ebb4a5c9f668362bc612a2e ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44904)
I20260929 02:46:43.812911 6565 tablet_copy_service.cc:161] P 61a426582b774c498b111e9c0b3c2df2: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9db8f5e40ebb4a5c9f668362bc612a2e at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44904: session id = 9db8f5e40ebb4a5c9f668362bc612a2e-00000000000000000000000000000000
I20260929 02:46:43.816262 6565 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:46:43.818203 7107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:46:43.825832 7107 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: tablet copy: Starting download of 0 data blocks...
I20260929 02:46:43.826170 7107 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: tablet copy: Starting download of 1 WAL segments...
I20260929 02:46:43.828944 7107 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:46:43.833745 6565 tablet_copy_service.cc:342] P 61a426582b774c498b111e9c0b3c2df2: Request end of tablet copy session 9db8f5e40ebb4a5c9f668362bc612a2e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:44904
I20260929 02:46:43.833909 6565 tablet_copy_service.cc:433] P 61a426582b774c498b111e9c0b3c2df2: ending tablet copy session 9db8f5e40ebb4a5c9f668362bc612a2e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9db8f5e40ebb4a5c9f668362bc612a2e
I20260929 02:46:43.844489 7107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:43.847499 7302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:43.847729 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 6606
W20260929 02:46:43.861322 7303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:43.867791 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:40727
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39317
--webserver_port=35571
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:43.869481 7305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:43.869652 7107 server_base.cc:1034] running on GCE node
I20260929 02:46:43.870081 7107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:43.871273 7107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:43.872612 7107 hybrid_clock.cc:648] HybridClock initialized: now 1790650003872221 us; error 342 us; skew 500 ppm
I20260929 02:46:43.872865 7107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:43 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650003, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:43.874704 7107 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260929 02:46:43.876863 7107 webserver.cc:460] Webserver started at http://127.0.97.60:35157/ using document root <none> and password file <none>
I20260929 02:46:43.877157 7107 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:43.877271 7107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:43.879741 7107 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 02:46:43.881754 7313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:43.882172 7107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 02:46:43.882309 7107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
uuid: "9db8f5e40ebb4a5c9f668362bc612a2e"
format_stamp: "Formatted at 2026-09-29 02:46:43 on dist-test-slave-6tjh"
server_key: "cf9f117d88af17ec5252cd48a46224c8"
server_key_iv: "9cda0721fc26fa49333ca11ac264e60b"
server_key_version: "encryptionkey@0"
I20260929 02:46:43.882494 7107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:43.896049 7107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:43.899801 7107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:43.925635 7107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41649
I20260929 02:46:43.925654 7364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41649 every 8 connection(s)
I20260929 02:46:43.926648 7107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
I20260929 02:46:43.933960 7365 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:46:43.939397 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Bootstrap starting.
I20260929 02:46:43.953118 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:46:43.953580 7365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: Bootstrap complete.
I20260929 02:46:43.954417 7365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } }
I20260929 02:46:43.954661 7365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9db8f5e40ebb4a5c9f668362bc612a2e, State: Initialized, Role: LEARNER
I20260929 02:46:43.954895 7365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } }
I20260929 02:46:43.955690 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:43.956022 7365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:43.956171 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: This master's current role is: LEARNER
I20260929 02:46:43.965199 7379 catalog_manager.cc:1259] Loaded cluster ID: c1a2063bb0094966bcca6788eea1c355
I20260929 02:46:43.965306 7379 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: loading cluster ID for follower catalog manager: success
I20260929 02:46:43.968761 7379 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: acquiring CA information for follower catalog manager: success
I20260929 02:46:43.971081 7379 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:46:44.238159 7310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:44.238432 7310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:44.238525 7310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:44.245667 7310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:44.245788 7310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:44.245901 7310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:44.256022 7310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:40727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35571
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:44.257400 7310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:44.258736 7310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:44.273007 7385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:44.273213 7384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:44.273155 7387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:44.274091 7310 server_base.cc:1034] running on GCE node
I20260929 02:46:44.274736 7310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:44.275751 7310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:44.276971 7310 hybrid_clock.cc:648] HybridClock initialized: now 1790650004276925 us; error 41 us; skew 500 ppm
I20260929 02:46:44.277357 7310 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:44 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:44.280877 7310 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 02:46:44.282594 7310 webserver.cc:460] Webserver started at http://127.0.97.1:35571/ using document root <none> and password file <none>
I20260929 02:46:44.283484 7310 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:44.283641 7310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:44.289736 7310 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:46:44.292500 7394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:44.294003 7310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:46:44.294190 7310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "ac8ff7c891f041c2b808157ebada8a5c"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f9ee871cd13e33ed64d751fdc91236fd"
server_key_iv: "eb7349fc103586aa8595badc0cd33798"
server_key_version: "encryptionkey@0"
I20260929 02:46:44.295619 7310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:44.314224 7310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:44.318888 7310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:44.319216 7310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:44.320437 7310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:44.322041 7310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:44.322153 7310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:44.322263 7310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:44.322341 7310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:44.326588 7399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.319232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.0.1:59342)
0929 02:46:44.319681 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:44.319701 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:44.319833 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:44.320374 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:44.320379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:44.320411 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:44.320669 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.320683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.322064 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.322070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.324431 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.324447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.325637 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.325643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.325887 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.325947 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 02:46:44.326046 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:46:44.326207 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":226,"threads_started":1}
I20260929 02:46:44.326653 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.319430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.0.97.60:41649)
0929 02:46:44.319985 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.319989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.320005 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:46:44.320063 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.320066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.320188 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.320342 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:44.320841 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.321900 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.324648 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.325447 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.325556 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.325707 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:44.326096 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 02:46:44.326243 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":306,"threads_started":1}
I20260929 02:46:44.327957 7340 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 02:46:44.328606 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61a426582b774c498b111e9c0b3c2df2. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:44.328892 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: This master's current role is: LEARNER
I20260929 02:46:44.360643 7310 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40727
I20260929 02:46:44.360700 7508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40727 every 8 connection(s)
I20260929 02:46:44.362309 7310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:44.367208 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 7310
I20260929 02:46:44.367607 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 6740
Sep 29 02:46:44 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 02:46:44 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 02:46:44 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260929 02:46:44.399154 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.366164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.97.1:34109)
0929 02:46:44.367523 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:44.367591 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:44.371067 (+ 3476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:44.372616 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:44.372634 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:44.373638 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:44.374675 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.374707 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.376693 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.376703 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.379667 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.379692 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.381540 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.381555 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.382043 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.382971 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:44.383016 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:44.387456 (+ 4440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:44.393132 (+ 5676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:44.393143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:44.393164 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:44.393854 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:44.394593 (+ 739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:44.394598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:44.394601 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:44.394919 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:44.396131 (+ 1212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:44.396141 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:44.396673 (+ 532us) client_negotiation.cc:769] Sending connection context
0929 02:46:44.397012 (+ 339us) client_negotiation.cc:240] Negotiation successful
0929 02:46:44.397629 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1021,"thread_start_us":781,"threads_started":1}
I20260929 02:46:44.400408 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.365612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34109 (local address 127.0.97.61:35441)
0929 02:46:44.366200 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.366209 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.367729 (+ 1520us) server_negotiation.cc:407] Connection header received
0929 02:46:44.371530 (+ 3801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.371540 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.371681 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.371784 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:44.374955 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.376392 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.379977 (+ 3585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.380894 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.381014 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.388305 (+ 7291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:44.388317 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:44.388327 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:44.388391 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:44.392950 (+ 4559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.394001 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.394007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.394010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.394468 (+ 458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.395077 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.395083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.395086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.395887 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:44.396052 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:44.399735 (+ 3683us) server_negotiation.cc:299] Negotiation successful
0929 02:46:44.399972 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":322,"threads_started":1}
I20260929 02:46:44.401715 7511 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35441
I20260929 02:46:44.401973 7511 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:44.402952 7511 heartbeater.cc:511] Master 127.0.97.61:35441 requested a full tablet report, sending...
I20260929 02:46:44.404007 7516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.368210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41455 (local address 127.0.97.62:39317)
0929 02:46:44.380794 (+ 12584us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.380804 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.381668 (+ 864us) server_negotiation.cc:407] Connection header received
0929 02:46:44.382041 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.382047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.382117 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.382195 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:44.384174 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.385416 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.387161 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.388240 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.388376 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.394524 (+ 6148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:44.394534 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:44.394540 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:44.394605 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:44.398450 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.400899 (+ 2449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.400911 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.400914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.401179 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.401581 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.401585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.401591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.402077 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:44.402207 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:44.403473 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0929 02:46:44.403694 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12142,"thread_start_us":12424,"threads_started":1}
Sep 29 02:46:44 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 02:46:44.404507 7515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.368013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.1:41455)
0929 02:46:44.381564 (+ 13551us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:44.381588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:44.381833 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:44.382906 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:44.382910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:44.383668 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:44.384008 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.384020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.385735 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.385739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.387014 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.387033 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.389621 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.389631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.389889 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.391225 (+ 1336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:44.391250 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:44.394290 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:44.398781 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:44.398788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:44.398793 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:44.400305 (+ 1512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:44.401314 (+ 1009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:44.401318 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:44.401321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:44.401459 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:44.402280 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:44.402285 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:44.402449 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 02:46:44.402593 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:46:44.402911 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13403,"thread_start_us":12961,"threads_started":1}
I20260929 02:46:44.405726 7510 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39317
I20260929 02:46:44.405881 7400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.365630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41861 (local address 127.0.97.60:41649)
0929 02:46:44.365832 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.365839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.384990 (+ 19151us) server_negotiation.cc:407] Connection header received
0929 02:46:44.387255 (+ 2265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.387262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.387344 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.387415 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:44.389047 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.390113 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.392296 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.393188 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.393301 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.397825 (+ 4524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:44.397833 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:44.397838 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:44.397882 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:44.401911 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.402911 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.402919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.402924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.403087 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.404239 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.404246 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.404252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.404648 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:44.404800 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:44.405345 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 02:46:44.405564 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:46:44.406280 7510 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:44.406354 7143 ts_manager.cc:194] Registered new tserver with Master: ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727)
I20260929 02:46:44.406744 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.370948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.97.1:41861)
0929 02:46:44.384865 (+ 13917us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:44.384890 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:44.385108 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:44.388046 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:44.388052 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:44.388672 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:44.388894 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.388902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.390281 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.390287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.392097 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.392117 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.393630 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.393636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:44.393930 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.394562 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:44.394586 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:44.397565 (+ 2979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:44.402093 (+ 4528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:44.402101 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:44.402106 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:44.402733 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:44.403255 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:44.403261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:44.403265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:44.404075 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:44.404899 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:44.404903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:44.405049 (+ 146us) client_negotiation.cc:769] Sending connection context
0929 02:46:44.406035 (+ 986us) client_negotiation.cc:240] Negotiation successful
0929 02:46:44.406337 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13778,"thread_start_us":11778,"threads_started":1}
I20260929 02:46:44.406868 7510 heartbeater.cc:511] Master 127.0.97.62:39317 requested a full tablet report, sending...
I20260929 02:46:44.408123 7509 heartbeater.cc:347] Connected to a master server at 127.0.97.60:41649
I20260929 02:46:44.408273 7509 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:44.408528 6545 ts_manager.cc:194] Re-registered known tserver with Master: ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727)
I20260929 02:46:44.408739 7509 heartbeater.cc:511] Master 127.0.97.60:41649 requested a full tablet report, sending...
I20260929 02:46:44.409816 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:41455
I20260929 02:46:44.410634 7330 ts_manager.cc:194] Registered new tserver with Master: ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727)
I20260929 02:46:44.414443 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.368998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52066 (local address 127.0.97.2:42933)
0929 02:46:44.385668 (+ 16670us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.385675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.389784 (+ 4109us) server_negotiation.cc:407] Connection header received
0929 02:46:44.389864 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.389874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.390163 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.390334 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:44.392243 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.394084 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.397045 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.398072 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.398218 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:44.406550 (+ 8332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:44.406569 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:44.406588 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:44.406638 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:44.411651 (+ 5013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.412371 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.412380 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.412384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.412516 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:44.412934 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:44.412939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:44.412942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:44.413250 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:44.413538 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:44.413844 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 02:46:44.414097 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16491,"thread_start_us":16539,"threads_started":1}
I20260929 02:46:44.415596 6863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:44.499994 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 6740
I20260929 02:46:44.515957 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42933
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39317
--webserver_port=37257
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:46:44.800858 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: true } } }
I20260929 02:46:44.801350 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:44.802564 7293 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: attempting to promote NON_VOTER 9db8f5e40ebb4a5c9f668362bc612a2e to VOTER
I20260929 02:46:44.803184 7293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } }
I20260929 02:46:44.805472 7152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Refusing update from remote peer 61a426582b774c498b111e9c0b3c2df2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:46:44.805464 7340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 LEARNER]: Refusing update from remote peer 61a426582b774c498b111e9c0b3c2df2: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 02:46:44.806063 7293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:46:44.806409 7522 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 02:46:44.809590 7522 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.810256 7152 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.813126 7340 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.816284 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.816581 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:44.816772 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.816963 7293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.817157 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:46:44.817193 7293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:44.817580 7368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.817847 7368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:46:44.818131 7522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "61a426582b774c498b111e9c0b3c2df2" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:44.818403 7522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2 [sys.catalog]: This master's current role is: LEADER
W20260929 02:46:44.853052 7521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:44.853384 7521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:44.853572 7521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:44.861187 7521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:44.861304 7521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:44.861438 7521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:44.871573 7521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=37257
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:44.873034 7521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:44.874349 7521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:44.888365 7533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:44.888432 7534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:44.888433 7536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:44.889289 7521 server_base.cc:1034] running on GCE node
I20260929 02:46:44.889918 7521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:44.890892 7521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:44.892088 7521 hybrid_clock.cc:648] HybridClock initialized: now 1790650004892042 us; error 38 us; skew 500 ppm
I20260929 02:46:44.892419 7521 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:44 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:44.895506 7521 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 02:46:44.897040 7521 webserver.cc:460] Webserver started at http://127.0.97.2:37257/ using document root <none> and password file <none>
I20260929 02:46:44.897716 7521 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:44.897855 7521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:44.903456 7521 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 02:46:44.906253 7543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:44.907667 7521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:46:44.907807 7521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "5036ef3454bc434ca0b479d99bb383cb"
format_stamp: "Formatted at 2026-09-29 02:46:41 on dist-test-slave-6tjh"
server_key: "f04d71c5b09f004ae71f59a71a9f6641"
server_key_iv: "b71d64b7aa87c4d3731b736abbdae47b"
server_key_version: "encryptionkey@0"
I20260929 02:46:44.908843 7521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:44.936023 7521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:44.940295 7521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:44.940599 7521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:44.941553 7521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:44.942826 7521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:44.942914 7521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:44.942983 7521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:44.943035 7521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:44.978330 7521 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42933
I20260929 02:46:44.978454 7655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42933 every 8 connection(s)
I20260929 02:46:44.980010 7521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:44.986923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 7521
I20260929 02:46:44.987263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 6873
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650004, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260929 02:46:45.017570 7660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.983085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38157 (local address 127.0.97.60:41649)
0929 02:46:44.983855 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.983860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.984885 (+ 1025us) server_negotiation.cc:407] Connection header received
0929 02:46:44.996599 (+ 11714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.996605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.996701 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.996795 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.000533 (+ 3738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.001771 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.005046 (+ 3275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.005825 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.005937 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.010387 (+ 4450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.010398 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.010406 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.010454 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.014304 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.015260 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.015265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.015269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.015437 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.015898 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.015903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.015907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.016227 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.016374 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.016922 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.017151 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":233,"threads_started":1}
I20260929 02:46:45.018918 7663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.983630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:35233)
0929 02:46:44.984768 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:44.984786 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:44.988577 (+ 3791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:44.993389 (+ 4812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:44.993398 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:44.994098 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:44.995172 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.995205 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.996922 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:44.996928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.000528 (+ 3600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.000563 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.001779 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.001786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.002619 (+ 833us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.003514 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:45.003551 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:45.007758 (+ 4207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:45.013605 (+ 5847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.013615 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.013635 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.014289 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.014828 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.014833 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.014837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.015116 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.015853 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:45.015866 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:45.016365 (+ 499us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.016721 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.017294 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":356,"spinlock_wait_cycles":26496,"thread_start_us":1242,"threads_started":1}
I20260929 02:46:45.018940 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.983630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.97.2:38157)
0929 02:46:44.984624 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:44.984739 (+ 115us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:44.996286 (+ 11547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:44.997089 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:44.997093 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:44.998245 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:44.998572 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:44.998594 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.002517 (+ 3923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.002523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.004837 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.004861 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.006282 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.006289 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.006475 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.007115 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:45.007145 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:45.010096 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:45.014460 (+ 4364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.014465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.014468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.015102 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.015591 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.015596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.015600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.015770 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.016471 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:45.016476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:45.016636 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.017875 (+ 1239us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.018175 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":552,"threads_started":1}
I20260929 02:46:45.020484 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.983602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35233 (local address 127.0.97.62:39317)
0929 02:46:44.986283 (+ 2681us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.986288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.986304 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:46:44.992755 (+ 6451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.992766 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.992868 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.992992 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:44.995455 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:44.996714 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.000776 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.001496 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.002123 (+ 627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.008039 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.008047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.008056 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.008104 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.013429 (+ 5325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.014475 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.014481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.014485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.014678 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.015276 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.015281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.015284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.015694 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.019646 (+ 3952us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.019800 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.020046 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2584,"thread_start_us":2472,"threads_started":1}
I20260929 02:46:45.021129 7656 heartbeater.cc:347] Connected to a master server at 127.0.97.60:41649
I20260929 02:46:45.021461 7656 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:45.021664 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.988320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.97.3:46157)
0929 02:46:44.995955 (+ 7635us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:44.995967 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:44.996004 (+ 37us) server_negotiation.cc:407] Connection header received
0929 02:46:44.996083 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:44.996094 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:44.996440 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:44.996654 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:44.999498 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.001616 (+ 2118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.004252 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.005229 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.005343 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.013594 (+ 8251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.013612 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.013633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.013692 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.017765 (+ 4073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.018657 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.018668 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.018673 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.018851 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.019302 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.019306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.019309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.019699 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.019989 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.020936 (+ 947us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.021275 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7410,"thread_start_us":7238,"threads_started":1}
I20260929 02:46:45.021878 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.986366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34447 (local address 127.0.97.61:35441)
0929 02:46:45.002337 (+ 15971us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.002343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.003127 (+ 784us) server_negotiation.cc:407] Connection header received
0929 02:46:45.003193 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.003199 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.003297 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.003404 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.005276 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.006418 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.008669 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.009405 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.009539 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.013795 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.013804 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.013811 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.013865 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.017952 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.018727 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.018733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.018736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.018873 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.019308 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.019313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.019316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.020257 (+ 941us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.020386 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.021304 (+ 918us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.021555 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15822,"thread_start_us":15648,"threads_started":1}
I20260929 02:46:45.022332 7656 heartbeater.cc:511] Master 127.0.97.60:41649 requested a full tablet report, sending...
I20260929 02:46:45.022571 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:44.985835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.97.2:34447)
0929 02:46:45.002844 (+ 17009us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.002868 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.002995 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.003589 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.003595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.004724 (+ 1129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:45.005082 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.005095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.006591 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.006595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.008456 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.008472 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.009778 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.009786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.010005 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.010568 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:45.010590 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:45.013592 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:45.018094 (+ 4502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.018100 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.018103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.018597 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.018996 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.019001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.019004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.019164 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.020436 (+ 1272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:45.020441 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:45.020617 (+ 176us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.020746 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.021018 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16860,"thread_start_us":16102,"threads_started":1}
I20260929 02:46:45.024386 7330 ts_manager.cc:194] Registered new tserver with Master: 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933)
I20260929 02:46:45.024816 7657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39317
I20260929 02:46:45.024881 7658 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35441
I20260929 02:46:45.024942 7657 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:45.025022 7658 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:45.025025 6996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:45.025413 7657 heartbeater.cc:511] Master 127.0.97.62:39317 requested a full tablet report, sending...
I20260929 02:46:45.025573 7658 heartbeater.cc:511] Master 127.0.97.61:35441 requested a full tablet report, sending...
I20260929 02:46:45.026422 6545 ts_manager.cc:194] Re-registered known tserver with Master: 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933)
I20260929 02:46:45.026710 7143 ts_manager.cc:194] Registered new tserver with Master: 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933)
I20260929 02:46:45.027590 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:35233
I20260929 02:46:45.111217 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 6873
I20260929 02:46:45.127789 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:46157
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39317
--webserver_port=36153
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:46:45.412205 7510 heartbeater.cc:503] Master 127.0.97.62:39317 was elected leader, sending a full tablet report...
W20260929 02:46:45.460264 7669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:45.460615 7669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:45.460773 7669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:45.468425 7669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:45.468531 7669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:45.468737 7669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:45.478808 7669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:46157
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=36153
--tserver_master_addrs=127.0.97.62:39317,127.0.97.61:35441,127.0.97.60:41649
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:45.480196 7669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:45.481568 7669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:45.495092 7674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:45.495051 7675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:45.495221 7677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:45.495919 7669 server_base.cc:1034] running on GCE node
I20260929 02:46:45.496480 7669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:45.497529 7669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:45.498730 7669 hybrid_clock.cc:648] HybridClock initialized: now 1790650005498693 us; error 39 us; skew 500 ppm
I20260929 02:46:45.499135 7669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:45.502391 7669 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 02:46:45.503796 7669 webserver.cc:460] Webserver started at http://127.0.97.3:36153/ using document root <none> and password file <none>
I20260929 02:46:45.504482 7669 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:45.504662 7669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:45.509832 7669 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:45.512701 7684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:45.514081 7669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:45.514253 7669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "b490e73593544d208b83ca28a1c91135"
format_stamp: "Formatted at 2026-09-29 02:46:42 on dist-test-slave-6tjh"
server_key: "a676e22965b0bdead07f87ded6b9e082"
server_key_iv: "38dae8746fcefb585fa4872a855b0c8a"
server_key_version: "encryptionkey@0"
I20260929 02:46:45.515203 7669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:45.540906 7669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:45.545483 7669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:45.545784 7669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:45.546785 7669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:45.548159 7669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:45.548278 7669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:45.548408 7669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:45.548496 7669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:45.582543 7669 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46157
I20260929 02:46:45.582635 7796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46157 every 8 connection(s)
I20260929 02:46:45.584205 7669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:45.588905 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 7669
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 29 02:46:45 dist-test-slave-6tjh krb5kdc[6500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650005, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:45.636082 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.587936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.97.3:56969)
0929 02:46:45.604525 (+ 16589us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.604606 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.608686 (+ 4080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.609791 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.609806 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.610837 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:45.612040 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.612070 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.613746 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.613752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.616462 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.616477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.617796 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.617804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.618234 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.619348 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:45.619377 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:45.624339 (+ 4962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:45.630253 (+ 5914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.630266 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.630288 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.631091 (+ 803us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.631566 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.631571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.631574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.631826 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.632644 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:45.632654 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:45.633174 (+ 520us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.633528 (+ 354us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.634271 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16267,"mutex_wait_us":364,"thread_start_us":17494,"threads_started":1}
I20260929 02:46:45.637696 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.587145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56969 (local address 127.0.97.60:41649)
0929 02:46:45.603066 (+ 15921us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.603075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.605403 (+ 2328us) server_negotiation.cc:407] Connection header received
0929 02:46:45.609020 (+ 3617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.609026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.609114 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.609222 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.612279 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.613506 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.616706 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.617403 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.617515 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.624771 (+ 7256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.624782 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.624791 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.624861 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.630072 (+ 5211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.631254 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.631260 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.631264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.631430 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.631971 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.631977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.631981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.632403 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.632566 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.636920 (+ 4354us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.637178 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15761,"thread_start_us":15930,"threads_started":1}
I20260929 02:46:45.638711 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.588692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.3:50189)
0929 02:46:45.605895 (+ 17203us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.605919 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.609095 (+ 3176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.609791 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.609806 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.611348 (+ 1542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:45.612040 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.612070 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.613746 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.613752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.616462 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.616477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.617796 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.617804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.618229 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.619584 (+ 1355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:45.625727 (+ 6143us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:45.630227 (+ 4500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:45.635768 (+ 5541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.635774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.635777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.636401 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.636936 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.636941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.636944 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.637077 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.637705 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:45.637713 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:45.637872 (+ 159us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.637999 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.638272 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17061,"mutex_wait_us":401,"spinlock_wait_cycles":20224,"thread_start_us":15420,"threads_started":1}
I20260929 02:46:45.638957 7797 heartbeater.cc:347] Connected to a master server at 127.0.97.60:41649
I20260929 02:46:45.639282 7797 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:45.639470 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.587154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50189 (local address 127.0.97.62:39317)
0929 02:46:45.602658 (+ 15504us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.602668 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.606109 (+ 3441us) server_negotiation.cc:407] Connection header received
0929 02:46:45.609250 (+ 3141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.609255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.609347 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.609491 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.612266 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.613471 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.616671 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.617297 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.617418 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.631251 (+ 13833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.631257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.631262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.631301 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.635635 (+ 4334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.636592 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.636601 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.636607 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.636815 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.637169 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.637177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.637183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.637588 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.638311 (+ 723us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.638501 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.639118 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15317,"thread_start_us":15192,"threads_started":1}
I20260929 02:46:45.640646 7797 heartbeater.cc:511] Master 127.0.97.60:41649 requested a full tablet report, sending...
I20260929 02:46:45.641856 7798 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39317
I20260929 02:46:45.641973 7798 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:45.642445 7798 heartbeater.cc:511] Master 127.0.97.62:39317 requested a full tablet report, sending...
I20260929 02:46:45.642535 7330 ts_manager.cc:194] Registered new tserver with Master: b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157)
I20260929 02:46:45.643592 6545 ts_manager.cc:194] Re-registered known tserver with Master: b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157)
I20260929 02:46:45.645008 6545 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50189
I20260929 02:46:45.648069 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.605379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.97.3:47919)
0929 02:46:45.608497 (+ 3118us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.608514 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.609924 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.610773 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.610782 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.611594 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:45.612084 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.612104 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.614079 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.614088 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.616566 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.616587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.618118 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.618126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.618347 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.619302 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:45.619348 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:45.630692 (+ 11344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:45.641948 (+ 11256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.641955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.641964 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.642514 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.645023 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:45.645032 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:45.645036 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:45.645216 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:45.646054 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:45.646060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:45.646215 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.646379 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.647609 (+ 1230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3003,"mutex_wait_us":91,"spinlock_wait_cycles":3840,"thread_start_us":3261,"threads_started":1}
I20260929 02:46:45.649276 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.605541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47919 (local address 127.0.97.61:35441)
0929 02:46:45.610413 (+ 4872us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.610419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.610437 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:46:45.610490 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.610494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.610568 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.610675 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.612347 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.613824 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.616803 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.617704 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.617863 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.631027 (+ 13164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.631038 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.631046 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.631120 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.641776 (+ 10656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.644507 (+ 2731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.644514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.644517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.644873 (+ 356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.645431 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.645436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.645439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.645843 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.645978 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.648517 (+ 2539us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.648801 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4732,"thread_start_us":4602,"threads_started":1}
I20260929 02:46:45.650199 7799 heartbeater.cc:347] Connected to a master server at 127.0.97.61:35441
I20260929 02:46:45.650352 7799 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:45.650790 7799 heartbeater.cc:511] Master 127.0.97.61:35441 requested a full tablet report, sending...
I20260929 02:46:45.652000 7143 ts_manager.cc:194] Registered new tserver with Master: b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157)
I20260929 02:46:45.661362 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.623113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.0.97.62:39317)
0929 02:46:45.639707 (+ 16594us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.639712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.639726 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:46:45.639779 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.639786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.639873 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.639964 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.646627 (+ 6663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.647319 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.650222 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.650857 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.650977 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.654903 (+ 3926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.654908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.654911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.654951 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.658247 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.659336 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.659342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.659346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.659507 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.659875 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.659881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.659884 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.660168 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.660410 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.660803 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.661016 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16500,"thread_start_us":22175,"threads_started":1}
I20260929 02:46:45.662999 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.619890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59354 (local address 127.0.97.60:41649)
0929 02:46:45.639071 (+ 19181us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.639077 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.639095 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:46:45.641121 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.641126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.641196 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.641290 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.644911 (+ 3621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.645882 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.650728 (+ 4846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.651389 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.651524 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.656375 (+ 4851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.656384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.656392 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.656432 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.659964 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.660899 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.660905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.660908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.661054 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.661413 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.661420 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.661423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.661898 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.662161 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.662510 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.662740 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19078,"thread_start_us":24460,"threads_started":1}
I20260929 02:46:45.666045 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.629822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.0.97.61:35441)
0929 02:46:45.640747 (+ 10925us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.640758 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.640778 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:46:45.645458 (+ 4680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.645462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.645536 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.645645 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.648356 (+ 2711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.650260 (+ 1904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.652460 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.653194 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.653329 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.656877 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.656884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.656889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.656929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.660790 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.662232 (+ 1442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.662238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.662241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.662437 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.664480 (+ 2043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.664485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.664487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.664808 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.665338 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.665457 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.665689 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10773,"thread_start_us":10954,"threads_started":1}
I20260929 02:46:45.665951 6545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52274:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:45.673666 6545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:45.714047 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.692067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40727 (local address 127.0.0.1:48682)
0929 02:46:45.704344 (+ 12277us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.704362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.704476 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.706696 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.706704 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.706738 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:45.707167 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.707180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.709274 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.709278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.711236 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.711251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.713028 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.713034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.713225 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.713258 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.713440 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.713580 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12127,"mutex_wait_us":20,"thread_start_us":11985,"threads_started":1}
I20260929 02:46:45.714555 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.694263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52080 (local address 127.0.97.2:42933)
0929 02:46:45.702964 (+ 8701us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.702975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.705144 (+ 2169us) server_negotiation.cc:407] Connection header received
0929 02:46:45.705315 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.705326 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.705629 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.705852 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:45.706504 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.709535 (+ 3031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.712256 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.713560 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.713702 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.713915 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.714129 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.714280 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8520,"thread_start_us":8710,"threads_started":1}
I20260929 02:46:45.714850 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.692331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48682 (local address 127.0.97.1:40727)
0929 02:46:45.706178 (+ 13847us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.706184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.706204 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:46:45.706282 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.706287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.706490 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.706688 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:45.707395 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.709111 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.711400 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.712824 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.712969 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.713193 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.714390 (+ 1197us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.714546 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13693,"thread_start_us":14663,"threads_started":1}
I20260929 02:46:45.715502 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.694022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42933 (local address 127.0.0.1:52080)
0929 02:46:45.705027 (+ 11005us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.705041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.705169 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.705862 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.705867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.705892 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:45.706323 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.706336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.709701 (+ 3365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.709706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.712104 (+ 2398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.712119 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.713755 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.713761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.713919 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.713954 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.714084 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.715169 (+ 1085us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10892,"mutex_wait_us":1,"thread_start_us":11367,"threads_started":1}
I20260929 02:46:45.717196 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.704420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46157 (local address 127.0.0.1:51396)
0929 02:46:45.705627 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.705643 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.705741 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.708167 (+ 2426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.708172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.708198 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:45.708644 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.708656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.711166 (+ 2510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.711171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.713138 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.713152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.715557 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.715565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.716586 (+ 1021us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.716617 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.716697 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.716840 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1102,"mutex_wait_us":23,"thread_start_us":367,"threads_started":1}
I20260929 02:46:45.717281 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.704718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.0.97.3:46157)
0929 02:46:45.707641 (+ 2923us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.707650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.707682 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:46:45.707763 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.707769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.707968 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.708820 (+ 852us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:45.709100 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.710620 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.713275 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.715336 (+ 2061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.715457 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.715809 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.716795 (+ 986us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.716956 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2742,"thread_start_us":2582,"threads_started":1}
I20260929 02:46:45.718848 7444 tablet_service.cc:1467] Processing CreateTablet for tablet ba07af7dafda40c1a298e42ab8b2578e (DEFAULT_TABLE table=test-table [id=aad0b670cc6944339e4059b0db928610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:45.719053 7591 tablet_service.cc:1467] Processing CreateTablet for tablet ba07af7dafda40c1a298e42ab8b2578e (DEFAULT_TABLE table=test-table [id=aad0b670cc6944339e4059b0db928610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:45.720209 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba07af7dafda40c1a298e42ab8b2578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:45.720243 7444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba07af7dafda40c1a298e42ab8b2578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:45.720882 7732 tablet_service.cc:1467] Processing CreateTablet for tablet ba07af7dafda40c1a298e42ab8b2578e (DEFAULT_TABLE table=test-table [id=aad0b670cc6944339e4059b0db928610]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:45.722163 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba07af7dafda40c1a298e42ab8b2578e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:45.737258 7831 tablet_bootstrap.cc:492] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Bootstrap starting.
I20260929 02:46:45.739784 7833 tablet_bootstrap.cc:492] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Bootstrap starting.
I20260929 02:46:45.740494 7831 tablet_bootstrap.cc:654] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:45.742714 7831 log.cc:824] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:45.743392 7833 tablet_bootstrap.cc:654] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:45.744587 7833 log.cc:824] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:45.745981 7832 tablet_bootstrap.cc:492] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Bootstrap starting.
I20260929 02:46:45.750200 7832 tablet_bootstrap.cc:654] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:45.751353 7832 log.cc:824] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:45.752537 7833 tablet_bootstrap.cc:492] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: No bootstrap required, opened a new log
I20260929 02:46:45.752876 7833 ts_tablet_manager.cc:1397] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260929 02:46:45.753364 7831 tablet_bootstrap.cc:492] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: No bootstrap required, opened a new log
I20260929 02:46:45.753696 7831 ts_tablet_manager.cc:1397] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260929 02:46:45.764365 7833 raft_consensus.cc:348] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.764992 7833 raft_consensus.cc:374] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:45.765092 7833 raft_consensus.cc:729] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b490e73593544d208b83ca28a1c91135, State: Initialized, Role: FOLLOWER
I20260929 02:46:45.765905 7833 consensus_queue.cc:260] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.768719 7833 ts_tablet_manager.cc:1428] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260929 02:46:45.769016 7798 heartbeater.cc:503] Master 127.0.97.62:39317 was elected leader, sending a full tablet report...
I20260929 02:46:45.773425 7832 tablet_bootstrap.cc:492] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: No bootstrap required, opened a new log
I20260929 02:46:45.773731 7832 ts_tablet_manager.cc:1397] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.005s
I20260929 02:46:45.773983 7831 raft_consensus.cc:348] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.774516 7831 raft_consensus.cc:374] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:45.774668 7831 raft_consensus.cc:729] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5036ef3454bc434ca0b479d99bb383cb, State: Initialized, Role: FOLLOWER
I20260929 02:46:45.775431 7831 consensus_queue.cc:260] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.778740 7657 heartbeater.cc:503] Master 127.0.97.62:39317 was elected leader, sending a full tablet report...
I20260929 02:46:45.779652 7831 ts_tablet_manager.cc:1428] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Time spent starting tablet: real 0.026s user 0.009s sys 0.007s
I20260929 02:46:45.784150 7832 raft_consensus.cc:348] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.784507 7832 raft_consensus.cc:374] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:45.784643 7832 raft_consensus.cc:729] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8ff7c891f041c2b808157ebada8a5c, State: Initialized, Role: FOLLOWER
I20260929 02:46:45.785151 7832 consensus_queue.cc:260] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.787223 7832 ts_tablet_manager.cc:1428] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260929 02:46:45.790544 7837 raft_consensus.cc:482] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:45.790875 7837 raft_consensus.cc:504] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.792860 7837 leader_election.cc:288] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727), 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933)
I20260929 02:46:45.800757 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.793572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47785 (local address 127.0.97.2:42933)
0929 02:46:45.793737 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.793741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.793754 (+ 13us) server_negotiation.cc:407] Connection header received
0929 02:46:45.793809 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.793812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.793863 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.793931 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:45.794551 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.795723 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.798496 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.799764 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.799916 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.800013 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.800332 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.800480 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:46:45.800799 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.793133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42933 (local address 127.0.97.3:47785)
0929 02:46:45.793336 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.793361 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.793496 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.793994 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.794000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.794034 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:45.794319 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.794329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.795883 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.795889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.798331 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.798338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.799994 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.800007 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.800166 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.800191 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.800277 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.800508 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":254}
I20260929 02:46:45.800908 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.793205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40727 (local address 127.0.97.3:47407)
0929 02:46:45.793378 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:45.793398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:45.793518 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:45.794043 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:45.794048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:45.794074 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:45.794288 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.794297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.795883 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.795889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.798323 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:45.798337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.800130 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.800134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:45.800350 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.800379 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:46:45.800468 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 02:46:45.800628 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":167,"spinlock_wait_cycles":2304}
I20260929 02:46:45.801311 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.793525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47407 (local address 127.0.97.1:40727)
0929 02:46:45.793734 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.793738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.793755 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:46:45.793808 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.793812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.793869 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.793994 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:45.794457 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.795724 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.798496 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.799853 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.799956 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.800037 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.800500 (+ 463us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.800696 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:46:45.801892 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07af7dafda40c1a298e42ab8b2578e" candidate_uuid: "b490e73593544d208b83ca28a1c91135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5036ef3454bc434ca0b479d99bb383cb" is_pre_election: true
I20260929 02:46:45.802145 7464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07af7dafda40c1a298e42ab8b2578e" candidate_uuid: "b490e73593544d208b83ca28a1c91135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8ff7c891f041c2b808157ebada8a5c" is_pre_election: true
I20260929 02:46:45.802496 7611 raft_consensus.cc:2454] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b490e73593544d208b83ca28a1c91135 in term 0.
I20260929 02:46:45.802645 7464 raft_consensus.cc:2454] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b490e73593544d208b83ca28a1c91135 in term 0.
I20260929 02:46:45.803192 7688 leader_election.cc:302] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5036ef3454bc434ca0b479d99bb383cb, b490e73593544d208b83ca28a1c91135; no voters:
I20260929 02:46:45.803550 7837 raft_consensus.cc:2781] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:45.803694 7837 raft_consensus.cc:482] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:45.803812 7837 raft_consensus.cc:3037] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:45.806435 7837 raft_consensus.cc:504] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.807056 7837 leader_election.cc:288] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [CANDIDATE]: Term 1 election: Requested vote from peers ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727), 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933)
I20260929 02:46:45.807380 7464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07af7dafda40c1a298e42ab8b2578e" candidate_uuid: "b490e73593544d208b83ca28a1c91135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8ff7c891f041c2b808157ebada8a5c"
I20260929 02:46:45.807559 7464 raft_consensus.cc:3037] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:45.807539 7611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba07af7dafda40c1a298e42ab8b2578e" candidate_uuid: "b490e73593544d208b83ca28a1c91135" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5036ef3454bc434ca0b479d99bb383cb"
I20260929 02:46:45.807737 7611 raft_consensus.cc:3037] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:45.809875 7464 raft_consensus.cc:2454] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b490e73593544d208b83ca28a1c91135 in term 1.
I20260929 02:46:45.810164 7611 raft_consensus.cc:2454] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b490e73593544d208b83ca28a1c91135 in term 1.
I20260929 02:46:45.810285 7685 leader_election.cc:302] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac8ff7c891f041c2b808157ebada8a5c, b490e73593544d208b83ca28a1c91135; no voters:
I20260929 02:46:45.810570 7837 raft_consensus.cc:2781] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:45.811013 7837 raft_consensus.cc:686] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 1 LEADER]: Becoming Leader. State: Replica: b490e73593544d208b83ca28a1c91135, State: Running, Role: LEADER
I20260929 02:46:45.811561 7837 consensus_queue.cc:237] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } }
I20260929 02:46:45.816732 6545 catalog_manager.cc:5495] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 reported cstate change: term changed from 0 to 1, leader changed from <none> to b490e73593544d208b83ca28a1c91135 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "b490e73593544d208b83ca28a1c91135" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:45.838639 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.833345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.0.97.3:46157)
0929 02:46:45.833624 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.833631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.833650 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:46:45.833915 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.833920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.834004 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.834155 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:45.834506 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.835209 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.836592 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.837007 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.837146 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.837602 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:45.838095 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:45.838250 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.838313 (+ 63us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.838386 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:46:45.846731 7611 raft_consensus.cc:1261] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 1 FOLLOWER]: Refusing update from remote peer b490e73593544d208b83ca28a1c91135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:45.846745 7464 raft_consensus.cc:1261] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Refusing update from remote peer b490e73593544d208b83ca28a1c91135: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:45.847755 7840 consensus_queue.cc:1035] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:45.848093 7837 consensus_queue.cc:1035] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:46:45.854261 7800 tablet.cc:2369] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:45.884653 6545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:45.884951 6545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52274:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:45.895144 6545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61a426582b774c498b111e9c0b3c2df2: Sending DeleteTablet for 3 replicas of tablet ba07af7dafda40c1a298e42ab8b2578e
I20260929 02:46:45.896649 7444 tablet_service.cc:1514] Processing DeleteTablet for tablet ba07af7dafda40c1a298e42ab8b2578e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48682
I20260929 02:46:45.896711 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet ba07af7dafda40c1a298e42ab8b2578e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52080
I20260929 02:46:45.902412 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 6513
I20260929 02:46:45.903596 6575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:45.904210 7732 tablet_service.cc:1514] Processing DeleteTablet for tablet ba07af7dafda40c1a298e42ab8b2578e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:45 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51396
I20260929 02:46:45.919610 7857 tablet_replica.cc:330] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: stopping tablet replica
I20260929 02:46:45.920395 7857 raft_consensus.cc:2229] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:45.921132 7853 tablet_replica.cc:330] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: stopping tablet replica
I20260929 02:46:45.921775 7854 tablet_replica.cc:330] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: stopping tablet replica
I20260929 02:46:45.922659 7857 raft_consensus.cc:2258] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:45.923622 7854 raft_consensus.cc:2229] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:45.924116 7853 raft_consensus.cc:2229] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:45.925138 7854 raft_consensus.cc:2258] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:45.925710 7853 raft_consensus.cc:2258] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:45.927898 7857 ts_tablet_manager.cc:1905] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:45.930441 7853 ts_tablet_manager.cc:1905] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:45.942754 7854 ts_tablet_manager.cc:1905] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:45.946614 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.899684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58200 (local address 127.0.97.61:35441)
0929 02:46:45.900729 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.900738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.900760 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:46:45.900843 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.900851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.900949 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.901081 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.913775 (+ 12694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.914930 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.917044 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.918277 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.918399 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.938502 (+ 20103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.938513 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.938519 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.938571 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.942936 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.943988 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.943995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.943999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.944156 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.944670 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.944676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.944681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.945056 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.945327 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.945989 (+ 662us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.946226 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":910}
I20260929 02:46:45.950315 7853 ts_tablet_manager.cc:1918] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:45.950480 7853 log.cc:1196] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/ba07af7dafda40c1a298e42ab8b2578e
I20260929 02:46:45.951030 7853 ts_tablet_manager.cc:1939] T ba07af7dafda40c1a298e42ab8b2578e P ac8ff7c891f041c2b808157ebada8a5c: Deleting consensus metadata
I20260929 02:46:45.951042 7857 ts_tablet_manager.cc:1918] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:45.951195 7857 log.cc:1196] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/ba07af7dafda40c1a298e42ab8b2578e
I20260929 02:46:45.951726 7857 ts_tablet_manager.cc:1939] T ba07af7dafda40c1a298e42ab8b2578e P b490e73593544d208b83ca28a1c91135: Deleting consensus metadata
I20260929 02:46:45.958374 7854 ts_tablet_manager.cc:1918] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:45.958563 7854 log.cc:1196] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/ba07af7dafda40c1a298e42ab8b2578e
I20260929 02:46:45.959131 7854 ts_tablet_manager.cc:1939] T ba07af7dafda40c1a298e42ab8b2578e P 5036ef3454bc434ca0b479d99bb383cb: Deleting consensus metadata
I20260929 02:46:45.967149 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:45.900428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.0.97.60:41649)
0929 02:46:45.900607 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:45.900614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:45.949166 (+ 48552us) server_negotiation.cc:407] Connection header received
0929 02:46:45.949527 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:45.949537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:45.949628 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:45.949718 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:45.953296 (+ 3578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.954023 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.955266 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:45.955932 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:45.956063 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:45.959640 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:45.959646 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:45.959652 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:45.959693 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:45.963329 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.964224 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.964230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.964233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.964395 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:45.965286 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:45.965293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:45.965299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:45.965760 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:45.966069 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:45.966415 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 02:46:45.966713 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:46:46.078186 6529 catalog_manager.cc:4841] TS b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157): tablet ba07af7dafda40c1a298e42ab8b2578e (table test-table [id=aad0b670cc6944339e4059b0db928610]) successfully deleted
I20260929 02:46:46.078207 6532 catalog_manager.cc:4841] TS 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933): tablet ba07af7dafda40c1a298e42ab8b2578e (table test-table [id=aad0b670cc6944339e4059b0db928610]) successfully deleted
I20260929 02:46:46.078543 6529 catalog_manager.cc:4841] TS ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727): tablet ba07af7dafda40c1a298e42ab8b2578e (table test-table [id=aad0b670cc6944339e4059b0db928610]) successfully deleted
I20260929 02:46:46.078826 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 6513
I20260929 02:46:46.123204 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:46.105606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.0.97.60:41649)
0929 02:46:46.105887 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:46.105897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:46.106794 (+ 897us) server_negotiation.cc:407] Connection header received
0929 02:46:46.106974 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:46.106982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:46.107091 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:46.107203 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:46.109406 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:46.110499 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:46.111811 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:46.112492 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:46.112700 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:46.116327 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:46.116334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:46.116340 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:46.116382 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:46.119890 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:46.120893 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:46.120899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:46.120903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:46.121034 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:46.121489 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:46.121493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:46.121495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:46.121872 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:46.122117 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:46.122628 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 02:46:46.122833 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:46:46.124143 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:46.107627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.0.97.61:35441)
0929 02:46:46.107816 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:46.107821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:46.108048 (+ 227us) server_negotiation.cc:407] Connection header received
0929 02:46:46.108240 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:46.108245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:46.108355 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:46.108462 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:46.110333 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:46.111432 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:46.112933 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:46.113582 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:46.113723 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:46.116845 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:46.116849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:46.116852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:46.116883 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:46.120126 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:46.121119 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:46.121126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:46.121129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:46.121293 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:46.121686 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:46.121691 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:46.121694 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:46.122030 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:46.122244 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:46.123698 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0929 02:46:46.123850 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 02:46:46.869234 7864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:46.868228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.3:50907)
0929 02:46:46.868758 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:46.868924 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":234,"threads_started":1}
W20260929 02:46:46.869639 7798 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39317 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:46.877084 7865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:46.876382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:49505)
0929 02:46:46.876795 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:46.876923 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":237,"threads_started":1}
W20260929 02:46:46.877468 7657 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39317 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:46.898988 7866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:46.898203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.1:60211)
0929 02:46:46.898701 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:46.898828 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
W20260929 02:46:46.899435 7510 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39317 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:47.395710 7867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61a426582b774c498b111e9c0b3c2df2)
I20260929 02:46:47.396006 7867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } }
I20260929 02:46:47.397493 7867 leader_election.cc:288] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61a426582b774c498b111e9c0b3c2df2 (127.0.97.62:39317), 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60:41649)
I20260929 02:46:47.398840 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.397754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52754)
0929 02:46:47.398351 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.398542 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":782,"threads_started":1}
W20260929 02:46:47.399103 7130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:47.399641 7871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 61a426582b774c498b111e9c0b3c2df2)
I20260929 02:46:47.399981 7871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } }
I20260929 02:46:47.400678 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.400262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52756)
0929 02:46:47.400412 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.400488 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:46:47.400887 7130 leader_election.cc:334] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61a426582b774c498b111e9c0b3c2df2 (127.0.97.62:39317): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:47.401727 7871 leader_election.cc:288] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61a426582b774c498b111e9c0b3c2df2 (127.0.97.62:39317), c22ecfc2606d4348b278ef212d1d20c8 (127.0.97.61:35441)
I20260929 02:46:47.403105 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.401952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52772)
0929 02:46:47.402555 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.402715 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":328,"threads_started":1}
W20260929 02:46:47.405249 7317 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:47.407215 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.406518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52786)
0929 02:46:47.406821 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.406974 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260929 02:46:47.407403 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.397755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41649 (local address 127.0.0.1:59384)
0929 02:46:47.398858 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.398877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:47.399537 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:47.400015 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:47.400021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:47.400056 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:47.400682 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.400695 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.402211 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.402216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.404781 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.404799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.406561 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.406567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.406736 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.406775 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:46:47.406885 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:46:47.407015 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":53,"thread_start_us":268,"threads_started":1}
I20260929 02:46:47.407459 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.398033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59384 (local address 127.0.97.60:41649)
0929 02:46:47.398592 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.398599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.399048 (+ 449us) server_negotiation.cc:407] Connection header received
0929 02:46:47.399763 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.399768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.399855 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.399971 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:47.400907 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.402068 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.404982 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.406414 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.406536 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.406630 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.406912 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.407046 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":318,"threads_started":1}
W20260929 02:46:47.407975 7317 leader_election.cc:334] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 61a426582b774c498b111e9c0b3c2df2 (127.0.97.62:39317): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:47.408138 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c22ecfc2606d4348b278ef212d1d20c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" is_pre_election: true
I20260929 02:46:47.408489 7340 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c22ecfc2606d4348b278ef212d1d20c8 in term 1.
I20260929 02:46:47.409214 7129 leader_election.cc:302] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9db8f5e40ebb4a5c9f668362bc612a2e, c22ecfc2606d4348b278ef212d1d20c8; no voters: 61a426582b774c498b111e9c0b3c2df2
I20260929 02:46:47.409699 7867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 02:46:47.409834 7867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 61a426582b774c498b111e9c0b3c2df2)
I20260929 02:46:47.409955 7867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:46:47.411628 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.402627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:35441 (local address 127.0.0.1:58222)
0929 02:46:47.403448 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.403470 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:47.403623 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:47.405364 (+ 1741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:47.405370 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:47.405404 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:47.405786 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.405798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.407399 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.407403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.409431 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.409442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.410950 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.410954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.411068 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.411088 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 02:46:47.411192 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 02:46:47.411323 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":2325,"threads_started":1}
I20260929 02:46:47.411691 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.402757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.0.97.61:35441)
0929 02:46:47.405072 (+ 2315us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.405077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.405100 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:46:47.405153 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.405157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.405222 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.405331 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:47.405995 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.407152 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.409604 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.410666 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.410755 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.410837 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.411216 (+ 379us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.411348 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2184,"thread_start_us":1993,"threads_started":1}
I20260929 02:46:47.412279 7152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c22ecfc2606d4348b278ef212d1d20c8" is_pre_election: true
I20260929 02:46:47.412563 7867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } }
I20260929 02:46:47.412985 7152 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9db8f5e40ebb4a5c9f668362bc612a2e in current term 2: Already voted for candidate c22ecfc2606d4348b278ef212d1d20c8 in this term.
I20260929 02:46:47.413280 7867 leader_election.cc:288] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [CANDIDATE]: Term 2 election: Requested vote from peers 61a426582b774c498b111e9c0b3c2df2 (127.0.97.62:39317), 9db8f5e40ebb4a5c9f668362bc612a2e (127.0.97.60:41649)
I20260929 02:46:47.413693 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c22ecfc2606d4348b278ef212d1d20c8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e"
I20260929 02:46:47.413820 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.413369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52808)
0929 02:46:47.413581 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.413674 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 02:46:47.413671 7317 leader_election.cc:302] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9db8f5e40ebb4a5c9f668362bc612a2e; no voters: 61a426582b774c498b111e9c0b3c2df2, c22ecfc2606d4348b278ef212d1d20c8
I20260929 02:46:47.413919 7340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:46:47.414669 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.414382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52818)
0929 02:46:47.414508 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.414565 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:46:47.414861 7130 leader_election.cc:334] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 61a426582b774c498b111e9c0b3c2df2 (127.0.97.62:39317): Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
I20260929 02:46:47.416141 7340 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c22ecfc2606d4348b278ef212d1d20c8 in term 2.
I20260929 02:46:47.416306 7871 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 02:46:47.416479 7129 leader_election.cc:302] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9db8f5e40ebb4a5c9f668362bc612a2e, c22ecfc2606d4348b278ef212d1d20c8; no voters: 61a426582b774c498b111e9c0b3c2df2
I20260929 02:46:47.416824 7867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:46:47.417266 7867 raft_consensus.cc:686] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [term 2 LEADER]: Becoming Leader. State: Replica: c22ecfc2606d4348b278ef212d1d20c8, State: Running, Role: LEADER
I20260929 02:46:47.417610 7867 consensus_queue.cc:237] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } }
I20260929 02:46:47.419484 7877 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c22ecfc2606d4348b278ef212d1d20c8. Latest consensus state: current_term: 2 leader_uuid: "c22ecfc2606d4348b278ef212d1d20c8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:47.419919 7877 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:47.420534 7879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:47.423933 7879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:47.424787 7879 catalog_manager.cc:1259] Loaded cluster ID: c1a2063bb0094966bcca6788eea1c355
I20260929 02:46:47.424875 7879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:47.425969 7879 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:47.426855 7879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Loaded TSK: 0
I20260929 02:46:47.427702 7879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:46:47.467274 7143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:47.474363 7143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:47.478756 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.478321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52836)
0929 02:46:47.478482 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.478581 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:46:47.478817 7340 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [term 2 FOLLOWER]: Refusing update from remote peer c22ecfc2606d4348b278ef212d1d20c8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 02:46:47.479655 7877 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 02:46:47.479916 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.479448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52852)
0929 02:46:47.479611 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.479721 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:46:47.480083 7130 consensus_peers.cc:487] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 -> Peer 61a426582b774c498b111e9c0b3c2df2 (127.0.97.62:39317): Couldn't send request to peer 61a426582b774c498b111e9c0b3c2df2. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 02:46:47.482376 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c22ecfc2606d4348b278ef212d1d20c8. Latest consensus state: current_term: 2 leader_uuid: "c22ecfc2606d4348b278ef212d1d20c8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:47.482754 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:46:47.484083 7877 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c22ecfc2606d4348b278ef212d1d20c8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:47.484400 7877 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:47.484723 7881 mvcc.cc:205] Tried to move back new op lower bound from 7334502430625640448 to 7334502430385704960. Current Snapshot: MvccSnapshot[applied={T|T < 7334502430625640448}]
I20260929 02:46:47.485394 7871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c22ecfc2606d4348b278ef212d1d20c8" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "61a426582b774c498b111e9c0b3c2df2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39317 } } peers { permanent_uuid: "c22ecfc2606d4348b278ef212d1d20c8" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 35441 } attrs { promote: false } } peers { permanent_uuid: "9db8f5e40ebb4a5c9f668362bc612a2e" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41649 } attrs { promote: false } } }
I20260929 02:46:47.485626 7871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9db8f5e40ebb4a5c9f668362bc612a2e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:46:47.503126 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.492583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46157 (local address 127.0.0.1:51424)
0929 02:46:47.492762 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.492785 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:47.492887 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:47.494269 (+ 1382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:47.494297 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:47.494332 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:47.494658 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.494672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.497208 (+ 2536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.497217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.499928 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.499949 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.502085 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.502095 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.502362 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.502418 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 02:46:47.502501 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 02:46:47.502694 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":33}
I20260929 02:46:47.504755 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.492875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.0.97.3:46157)
0929 02:46:47.493847 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.493855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.493874 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:46:47.493960 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.493968 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.494066 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.495378 (+ 1312us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:47.495591 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.496998 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.500165 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.501852 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.504028 (+ 2176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.504200 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.504261 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.504442 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":507,"threads_started":1}
I20260929 02:46:47.505404 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.492926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42933 (local address 127.0.0.1:52084)
0929 02:46:47.493234 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.493251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:47.493358 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:47.494320 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:47.494329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:47.494363 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:47.495171 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.495200 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.497820 (+ 2620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.497828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.500707 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.500727 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.504264 (+ 3537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.504274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.504575 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.504631 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 02:46:47.504712 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 02:46:47.504946 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":357}
I20260929 02:46:47.507506 7732 tablet_service.cc:1467] Processing CreateTablet for tablet 18707f3211cf49eaa2a480dd869e8ca5 (DEFAULT_TABLE table=test-table [id=ecb1a0deb49145ab88d19610a40fef07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:47.507627 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.493273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.0.97.2:42933)
0929 02:46:47.493890 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.493899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.493924 (+ 25us) server_negotiation.cc:407] Connection header received
0929 02:46:47.494010 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.494019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.494117 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.494637 (+ 520us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:47.495413 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.496879 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.502203 (+ 5324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.503955 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.507044 (+ 3089us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.507148 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.507197 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.507354 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":526,"threads_started":1}
I20260929 02:46:47.507959 7732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18707f3211cf49eaa2a480dd869e8ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:47.509184 7591 tablet_service.cc:1467] Processing CreateTablet for tablet 18707f3211cf49eaa2a480dd869e8ca5 (DEFAULT_TABLE table=test-table [id=ecb1a0deb49145ab88d19610a40fef07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:47.509620 7591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18707f3211cf49eaa2a480dd869e8ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:47.510154 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.493651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40727 (local address 127.0.0.1:48684)
0929 02:46:47.503688 (+ 10037us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.503710 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:47.503831 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:47.505158 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:47.505164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:47.505191 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:47.505341 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.505348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.506578 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.506585 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.508081 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.508093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.509349 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.509356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.509546 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.509618 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 02:46:47.509742 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 02:46:47.509866 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9891,"thread_start_us":13066,"threads_started":1}
I20260929 02:46:47.510376 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.493762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.0.97.1:40727)
0929 02:46:47.504720 (+ 10958us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.504729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.504752 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:46:47.504825 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.504830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.504912 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.505457 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:47.505656 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.506457 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.508233 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.509196 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.509428 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.509537 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.509805 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.509932 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10825,"thread_start_us":11231,"threads_started":1}
I20260929 02:46:47.510987 7444 tablet_service.cc:1467] Processing CreateTablet for tablet 18707f3211cf49eaa2a480dd869e8ca5 (DEFAULT_TABLE table=test-table [id=ecb1a0deb49145ab88d19610a40fef07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:47.511374 7444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18707f3211cf49eaa2a480dd869e8ca5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:47.514468 7890 tablet_bootstrap.cc:492] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: Bootstrap starting.
I20260929 02:46:47.515925 7891 tablet_bootstrap.cc:492] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: Bootstrap starting.
I20260929 02:46:47.516350 7890 tablet_bootstrap.cc:654] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:47.517584 7891 tablet_bootstrap.cc:654] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:47.518996 7890 tablet_bootstrap.cc:492] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: No bootstrap required, opened a new log
I20260929 02:46:47.519148 7890 ts_tablet_manager.cc:1397] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:46:47.519618 7891 tablet_bootstrap.cc:492] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: No bootstrap required, opened a new log
I20260929 02:46:47.519773 7891 ts_tablet_manager.cc:1397] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:46:47.519884 7890 raft_consensus.cc:348] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.520077 7890 raft_consensus.cc:374] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:47.520179 7890 raft_consensus.cc:729] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b490e73593544d208b83ca28a1c91135, State: Initialized, Role: FOLLOWER
I20260929 02:46:47.520429 7890 consensus_queue.cc:260] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.520493 7891 raft_consensus.cc:348] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.520737 7892 tablet_bootstrap.cc:492] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Bootstrap starting.
I20260929 02:46:47.520752 7891 raft_consensus.cc:374] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:47.520843 7891 raft_consensus.cc:729] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5036ef3454bc434ca0b479d99bb383cb, State: Initialized, Role: FOLLOWER
I20260929 02:46:47.521062 7891 consensus_queue.cc:260] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.521807 7891 ts_tablet_manager.cc:1428] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:47.523034 7892 tablet_bootstrap.cc:654] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:47.527314 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.523187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:51865)
0929 02:46:47.526878 (+ 3691us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.527027 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3534,"thread_start_us":3386,"threads_started":1}
I20260929 02:46:47.528108 7890 ts_tablet_manager.cc:1428] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260929 02:46:47.529917 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.528749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.3:41835)
0929 02:46:47.529424 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.529586 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":371,"threads_started":1}
I20260929 02:46:47.531031 7658 heartbeater.cc:503] Master 127.0.97.61:35441 was elected leader, sending a full tablet report...
I20260929 02:46:47.531760 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.530750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:35289)
0929 02:46:47.531513 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.531600 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":647}
I20260929 02:46:47.532186 7799 heartbeater.cc:503] Master 127.0.97.61:35441 was elected leader, sending a full tablet report...
I20260929 02:46:47.532292 7892 tablet_bootstrap.cc:492] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: No bootstrap required, opened a new log
I20260929 02:46:47.532476 7892 ts_tablet_manager.cc:1397] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260929 02:46:47.533416 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.533037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.3:50755)
0929 02:46:47.533188 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.533272 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:46:47.533365 7892 raft_consensus.cc:348] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.533617 7892 raft_consensus.cc:374] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:47.533728 7892 raft_consensus.cc:729] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8ff7c891f041c2b808157ebada8a5c, State: Initialized, Role: FOLLOWER
I20260929 02:46:47.533993 7892 consensus_queue.cc:260] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.534744 7892 ts_tablet_manager.cc:1428] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:46:47.536003 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.535043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.1:34519)
0929 02:46:47.535582 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.535728 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":297,"threads_started":1}
I20260929 02:46:47.536998 7511 heartbeater.cc:503] Master 127.0.97.61:35441 was elected leader, sending a full tablet report...
I20260929 02:46:47.537725 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.537030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.1:57339)
0929 02:46:47.537504 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.537585 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260929 02:46:47.585220 7896 raft_consensus.cc:482] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:47.585497 7896 raft_consensus.cc:504] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.587172 7896 leader_election.cc:288] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157), ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727)
I20260929 02:46:47.595012 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.588158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58909 (local address 127.0.97.1:40727)
0929 02:46:47.588365 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.588372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.588898 (+ 526us) server_negotiation.cc:407] Connection header received
0929 02:46:47.589011 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.589016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.589092 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.589187 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:47.589736 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.590409 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.593306 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.594093 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.594235 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.594347 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.594625 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.594765 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:46:47.595111 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.587489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46157 (local address 127.0.97.2:58967)
0929 02:46:47.588065 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.588085 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:47.588289 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:47.588878 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:47.588883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:47.588920 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:47.589214 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.589223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.590658 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.590664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.593150 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.593164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.594385 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.594394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.594540 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.594597 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 02:46:47.594689 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 02:46:47.594816 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":124,"spinlock_wait_cycles":4992}
I20260929 02:46:47.595244 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.587850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40727 (local address 127.0.97.2:58909)
0929 02:46:47.588782 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.588800 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:47.588905 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:47.589230 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:47.589234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:47.589258 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:47.589575 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.589583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.590531 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.590539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.593152 (+ 2613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:47.593164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.594283 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.594288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:47.594471 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.594494 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:46:47.594569 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 02:46:47.594677 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":167,"thread_start_us":687,"threads_started":1}
I20260929 02:46:47.595518 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.587675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58967 (local address 127.0.97.3:46157)
0929 02:46:47.587911 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.587917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.588170 (+ 253us) server_negotiation.cc:407] Connection header received
0929 02:46:47.588463 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.588472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.588598 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.588665 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:47.589444 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.590532 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.593357 (+ 2825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.594119 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.594273 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.594362 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.594736 (+ 374us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.594874 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:46:47.595855 7464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18707f3211cf49eaa2a480dd869e8ca5" candidate_uuid: "5036ef3454bc434ca0b479d99bb383cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8ff7c891f041c2b808157ebada8a5c" is_pre_election: true
I20260929 02:46:47.596093 7464 raft_consensus.cc:2454] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5036ef3454bc434ca0b479d99bb383cb in term 0.
I20260929 02:46:47.596271 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18707f3211cf49eaa2a480dd869e8ca5" candidate_uuid: "5036ef3454bc434ca0b479d99bb383cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b490e73593544d208b83ca28a1c91135" is_pre_election: true
I20260929 02:46:47.596725 7752 raft_consensus.cc:2454] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5036ef3454bc434ca0b479d99bb383cb in term 0.
I20260929 02:46:47.596679 7544 leader_election.cc:302] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5036ef3454bc434ca0b479d99bb383cb, ac8ff7c891f041c2b808157ebada8a5c; no voters:
I20260929 02:46:47.597093 7896 raft_consensus.cc:2781] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:47.597223 7896 raft_consensus.cc:482] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:47.597349 7896 raft_consensus.cc:3037] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:47.599869 7896 raft_consensus.cc:504] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.600505 7896 leader_election.cc:288] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [CANDIDATE]: Term 1 election: Requested vote from peers b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157), ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727)
I20260929 02:46:47.600962 7752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18707f3211cf49eaa2a480dd869e8ca5" candidate_uuid: "5036ef3454bc434ca0b479d99bb383cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b490e73593544d208b83ca28a1c91135"
I20260929 02:46:47.601146 7752 raft_consensus.cc:3037] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:47.601096 7464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18707f3211cf49eaa2a480dd869e8ca5" candidate_uuid: "5036ef3454bc434ca0b479d99bb383cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac8ff7c891f041c2b808157ebada8a5c"
I20260929 02:46:47.601325 7464 raft_consensus.cc:3037] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:47.603397 7752 raft_consensus.cc:2454] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5036ef3454bc434ca0b479d99bb383cb in term 1.
I20260929 02:46:47.603461 7464 raft_consensus.cc:2454] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5036ef3454bc434ca0b479d99bb383cb in term 1.
I20260929 02:46:47.603787 7544 leader_election.cc:302] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5036ef3454bc434ca0b479d99bb383cb, b490e73593544d208b83ca28a1c91135; no voters:
I20260929 02:46:47.604040 7896 raft_consensus.cc:2781] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:47.604486 7896 raft_consensus.cc:686] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 1 LEADER]: Becoming Leader. State: Replica: 5036ef3454bc434ca0b479d99bb383cb, State: Running, Role: LEADER
I20260929 02:46:47.605176 7896 consensus_queue.cc:237] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } }
I20260929 02:46:47.605543 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.605117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:44055)
0929 02:46:47.605275 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.605379 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:46:47.609787 7142 catalog_manager.cc:5495] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5036ef3454bc434ca0b479d99bb383cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5036ef3454bc434ca0b479d99bb383cb" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 } health_report { overall_health: UNKNOWN } } }
W20260929 02:46:47.617956 7512 tablet.cc:2369] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:47.635572 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.630570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52098 (local address 127.0.97.2:42933)
0929 02:46:47.630794 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.630799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.630815 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:46:47.630907 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.630911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.630989 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.631112 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:47.631460 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.632171 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.633508 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.633945 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.634059 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.634434 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:47.635005 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:47.635112 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.635294 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.635365 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:46:47.639950 7464 raft_consensus.cc:1261] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Refusing update from remote peer 5036ef3454bc434ca0b479d99bb383cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:47.640364 7752 raft_consensus.cc:1261] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 1 FOLLOWER]: Refusing update from remote peer 5036ef3454bc434ca0b479d99bb383cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:47.640726 7903 consensus_queue.cc:1035] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac8ff7c891f041c2b808157ebada8a5c" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:47.641096 7896 consensus_queue.cc:1035] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b490e73593544d208b83ca28a1c91135" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:47.643898 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.643350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.3:44763)
0929 02:46:47.643513 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.643655 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 02:46:47.644219 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.643719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.1:37081)
0929 02:46:47.643935 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.644030 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 02:46:47.650391 7906 mvcc.cc:205] Tried to move back new op lower bound from 7334502431286439936 to 7334502431154573312. Current Snapshot: MvccSnapshot[applied={T|T < 7334502431286439936}]
I20260929 02:46:47.651543 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.650816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:41823)
0929 02:46:47.651194 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.651307 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195}
I20260929 02:46:47.655241 7897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.654660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.2:56979)
0929 02:46:47.654979 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.655073 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260929 02:46:47.655228 7907 mvcc.cc:205] Tried to move back new op lower bound from 7334502431286439936 to 7334502431154573312. Current Snapshot: MvccSnapshot[applied={T|T < 7334502431286439936}]
I20260929 02:46:47.655493 7909 mvcc.cc:205] Tried to move back new op lower bound from 7334502431286439936 to 7334502431154573312. Current Snapshot: MvccSnapshot[applied={T|T < 7334502431286439936}]
I20260929 02:46:47.656790 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.656158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.1:36889)
0929 02:46:47.656390 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.656536 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 02:46:47.657712 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.657068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.97.3:60501)
0929 02:46:47.657364 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.657482 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 02:46:47.671787 7143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:47.672003 7143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:47.677595 7143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c22ecfc2606d4348b278ef212d1d20c8: Sending DeleteTablet for 3 replicas of tablet 18707f3211cf49eaa2a480dd869e8ca5
I20260929 02:46:47.678673 7732 tablet_service.cc:1514] Processing DeleteTablet for tablet 18707f3211cf49eaa2a480dd869e8ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:47 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:51424
I20260929 02:46:47.679126 7591 tablet_service.cc:1514] Processing DeleteTablet for tablet 18707f3211cf49eaa2a480dd869e8ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:47 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:52084
I20260929 02:46:47.679167 7444 tablet_service.cc:1514] Processing DeleteTablet for tablet 18707f3211cf49eaa2a480dd869e8ca5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:47 UTC) from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:48684
I20260929 02:46:47.682160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 7310
I20260929 02:46:47.691203 7917 tablet_replica.cc:330] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: stopping tablet replica
I20260929 02:46:47.691592 7917 raft_consensus.cc:2229] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:47.692402 7917 raft_consensus.cc:2258] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:47.693714 7917 ts_tablet_manager.cc:1905] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:47.701267 7917 ts_tablet_manager.cc:1918] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:47.701642 7917 log.cc:1196] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/18707f3211cf49eaa2a480dd869e8ca5
I20260929 02:46:47.702235 7917 ts_tablet_manager.cc:1939] T 18707f3211cf49eaa2a480dd869e8ca5 P 5036ef3454bc434ca0b479d99bb383cb: Deleting consensus metadata
I20260929 02:46:47.703712 7130 catalog_manager.cc:4841] TS 5036ef3454bc434ca0b479d99bb383cb (127.0.97.2:42933): tablet 18707f3211cf49eaa2a480dd869e8ca5 (table test-table [id=ecb1a0deb49145ab88d19610a40fef07]) successfully deleted
I20260929 02:46:47.706362 7918 tablet_replica.cc:330] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: stopping tablet replica
I20260929 02:46:47.706715 7918 raft_consensus.cc:2229] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:47.707127 7918 raft_consensus.cc:2258] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:47.707778 7916 tablet_replica.cc:330] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: stopping tablet replica
I20260929 02:46:47.708539 7918 ts_tablet_manager.cc:1905] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:47.708788 7916 raft_consensus.cc:2229] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:47.709110 7916 raft_consensus.cc:2258] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:47.710361 7916 ts_tablet_manager.cc:1905] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:47.715965 7918 ts_tablet_manager.cc:1918] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:47.716122 7918 log.cc:1196] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/18707f3211cf49eaa2a480dd869e8ca5
I20260929 02:46:47.716675 7918 ts_tablet_manager.cc:1939] T 18707f3211cf49eaa2a480dd869e8ca5 P ac8ff7c891f041c2b808157ebada8a5c: Deleting consensus metadata
I20260929 02:46:47.717481 7916 ts_tablet_manager.cc:1918] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:47.717631 7916 log.cc:1196] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/18707f3211cf49eaa2a480dd869e8ca5
I20260929 02:46:47.717998 7127 catalog_manager.cc:4841] TS ac8ff7c891f041c2b808157ebada8a5c (127.0.97.1:40727): tablet 18707f3211cf49eaa2a480dd869e8ca5 (table test-table [id=ecb1a0deb49145ab88d19610a40fef07]) successfully deleted
I20260929 02:46:47.718115 7916 ts_tablet_manager.cc:1939] T 18707f3211cf49eaa2a480dd869e8ca5 P b490e73593544d208b83ca28a1c91135: Deleting consensus metadata
I20260929 02:46:47.719340 7127 catalog_manager.cc:4841] TS b490e73593544d208b83ca28a1c91135 (127.0.97.3:46157): tablet 18707f3211cf49eaa2a480dd869e8ca5 (table test-table [id=ecb1a0deb49145ab88d19610a40fef07]) successfully deleted
I20260929 02:46:47.731541 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.684174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.0.97.1:40727)
0929 02:46:47.684353 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.684362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.704430 (+ 20068us) server_negotiation.cc:407] Connection header received
0929 02:46:47.704693 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.704699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.704794 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.704913 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:47.707981 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.709183 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.718566 (+ 9383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.719229 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.719390 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.722188 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:47.722205 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:47.722231 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:47.722265 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:47.726726 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:47.727858 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:47.727904 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:47.727911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:47.728198 (+ 287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:47.728872 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:47.728878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:47.728882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:47.729362 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:47.729696 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.730890 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.731143 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 02:46:47.738420 7504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:47.852083 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 7310
I20260929 02:46:47.870265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 7521
I20260929 02:46:47.889911 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.871034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.0.97.2:42933)
0929 02:46:47.871223 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:47.871229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:47.871336 (+ 107us) server_negotiation.cc:407] Connection header received
0929 02:46:47.871563 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:47.871571 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:47.871635 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:47.871754 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:47.873099 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.873916 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.875111 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:47.875524 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:47.875659 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:47.883853 (+ 8194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:47.883866 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:47.883881 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:47.883923 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:47.887023 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:47.887703 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:47.887717 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:47.887719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:47.887896 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:47.888291 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:47.888297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:47.888301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:47.888690 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:47.889051 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:47.889294 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 02:46:47.889525 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:46:47.890679 7651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:47.996846 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.996332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52856)
0929 02:46:47.996519 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.996664 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:46:47.998330 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:47.997902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39317 (local address 127.0.0.1:52872)
0929 02:46:47.998068 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:47.998175 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:39317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 02:46:48.013032 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 7521
I20260929 02:46:48.032470 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 7669
I20260929 02:46:48.045962 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:48.033287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.0.97.3:46157)
0929 02:46:48.033471 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:48.033479 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:48.033540 (+ 61us) server_negotiation.cc:407] Connection header received
0929 02:46:48.033745 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:48.033751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:48.033836 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:48.033944 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:48.035291 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:48.035995 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:48.037332 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:48.037739 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:48.037867 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:48.040478 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:48.040492 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:48.040508 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:48.040540 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:48.043173 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:48.043843 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:48.043854 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:48.043862 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:48.043991 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:48.044378 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:48.044385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:48.044388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:48.044824 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:48.045130 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:48.045431 (+ 301us) server_negotiation.cc:299] Negotiation successful
0929 02:46:48.045646 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:46:48.046787 7792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:48.165683 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 7669
I20260929 02:46:48.187479 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 7006
I20260929 02:46:48.188625 7173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:48.308528 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 7006
I20260929 02:46:48.329150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 7107
I20260929 02:46:48.330164 7360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:48.445467 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 7107
2026-09-29T02:46:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9472 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:48 dist-test-slave-6tjh krb5kdc[7928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:48 dist-test-slave-6tjh krb5kdc[7928](info): set up 2 sockets
Sep 29 02:46:48 dist-test-slave-6tjh krb5kdc[7928](info): commencing operation
krb5kdc: starting...
W20260929 02:46:49.573611 388 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:49 dist-test-slave-6tjh krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260929 02:46:49.800206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36871
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:50.131845 7941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:50.132148 7941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:50.132299 7941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:50.139849 7941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:50.139971 7941 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 02:46:50.140048 7941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:50.140161 7941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:50.140254 7941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:50.140359 7941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:50.150733 7941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36871
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:50.152101 7941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:50.153522 7941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:50.165673 7946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:50.165719 7949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:50.165696 7947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:50.166694 7941 server_base.cc:1034] running on GCE node
I20260929 02:46:50.167577 7941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:50.169158 7941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:50.170433 7941 hybrid_clock.cc:648] HybridClock initialized: now 1790650010170383 us; error 46 us; skew 500 ppm
I20260929 02:46:50.170809 7941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:50 dist-test-slave-6tjh krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650010, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:50.174253 7941 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260929 02:46:50.175781 7941 webserver.cc:460] Webserver started at http://127.0.97.62:38005/ using document root <none> and password file <none>
I20260929 02:46:50.176600 7941 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:50.176735 7941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:50.177083 7941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:50.180123 7941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "d41e939183ad4d4b808384c85c9e8fe3"
format_stamp: "Formatted at 2026-09-29 02:46:50 on dist-test-slave-6tjh"
server_key: "2ff451a1ed660c745b93fdd149a231f1"
server_key_iv: "a54d9b1a596fd70c87f081812e696855"
server_key_version: "encryptionkey@0"
I20260929 02:46:50.181053 7941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "d41e939183ad4d4b808384c85c9e8fe3"
format_stamp: "Formatted at 2026-09-29 02:46:50 on dist-test-slave-6tjh"
server_key: "2ff451a1ed660c745b93fdd149a231f1"
server_key_iv: "a54d9b1a596fd70c87f081812e696855"
server_key_version: "encryptionkey@0"
I20260929 02:46:50.186406 7941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:46:50.190060 7956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:50.191483 7941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 02:46:50.191689 7941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "d41e939183ad4d4b808384c85c9e8fe3"
format_stamp: "Formatted at 2026-09-29 02:46:50 on dist-test-slave-6tjh"
server_key: "2ff451a1ed660c745b93fdd149a231f1"
server_key_iv: "a54d9b1a596fd70c87f081812e696855"
server_key_version: "encryptionkey@0"
I20260929 02:46:50.191921 7941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:50.218252 7941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:50.223004 7941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:50.223285 7941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:50.244957 7941 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36871
I20260929 02:46:50.244964 8017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36871 every 8 connection(s)
I20260929 02:46:50.246392 7941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:50.250653 8018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:50.251772 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 7941
I20260929 02:46:50.251979 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:50.252719 388 external_mini_cluster.cc:1428] Setting key 05de7b8bc74c265e71b9d7fb63881bdb
I20260929 02:46:50.264344 8018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Bootstrap starting.
Sep 29 02:46:50 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:50.267438 8018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:50.268697 8018 log.cc:824] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:50.271551 8018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: No bootstrap required, opened a new log
I20260929 02:46:50.275174 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:50.254813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59132 (local address 127.0.97.62:36871)
0929 02:46:50.255626 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:50.255639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:50.255688 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:46:50.257330 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:50.257353 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:50.257749 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:50.258078 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:50.259834 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.261231 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.262978 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.263611 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.263739 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:50.267662 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:50.267669 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:50.267684 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:50.267727 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:50.270879 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:50.271625 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:50.271632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:50.271638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:50.271860 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:50.272251 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:50.272261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:50.272265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:50.272853 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:50.273159 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:50.273592 (+ 433us) server_negotiation.cc:299] Negotiation successful
0929 02:46:50.274123 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":404,"threads_started":1}
I20260929 02:46:50.279506 8018 raft_consensus.cc:348] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e939183ad4d4b808384c85c9e8fe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260929 02:46:50.279872 8018 raft_consensus.cc:374] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:50.279979 8018 raft_consensus.cc:729] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41e939183ad4d4b808384c85c9e8fe3, State: Initialized, Role: FOLLOWER
I20260929 02:46:50.280503 8018 consensus_queue.cc:260] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e939183ad4d4b808384c85c9e8fe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260929 02:46:50.280823 8018 raft_consensus.cc:388] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:50.280992 8018 raft_consensus.cc:482] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:50.281175 8018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:50.283918 8018 raft_consensus.cc:504] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e939183ad4d4b808384c85c9e8fe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260929 02:46:50.284485 8018 leader_election.cc:302] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d41e939183ad4d4b808384c85c9e8fe3; no voters:
I20260929 02:46:50.285215 8018 leader_election.cc:288] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:50.285362 8023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:50.286636 8023 raft_consensus.cc:686] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [term 1 LEADER]: Becoming Leader. State: Replica: d41e939183ad4d4b808384c85c9e8fe3, State: Running, Role: LEADER
I20260929 02:46:50.287201 8023 consensus_queue.cc:237] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e939183ad4d4b808384c85c9e8fe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } }
I20260929 02:46:50.287685 8018 sys_catalog.cc:564] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:50.290297 8025 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d41e939183ad4d4b808384c85c9e8fe3. Latest consensus state: current_term: 1 leader_uuid: "d41e939183ad4d4b808384c85c9e8fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e939183ad4d4b808384c85c9e8fe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } } }
I20260929 02:46:50.290516 8024 sys_catalog.cc:455] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d41e939183ad4d4b808384c85c9e8fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d41e939183ad4d4b808384c85c9e8fe3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36871 } } }
I20260929 02:46:50.290783 8025 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:50.290823 8024 sys_catalog.cc:458] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:50.293044 8031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:50.297849 8031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:50.307211 8031 catalog_manager.cc:1347] Generated new cluster ID: 2729fd9a5cb042df8ed17ee80f42fc3d
I20260929 02:46:50.307375 8031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:50.342847 8031 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:50.344890 8031 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:50.365154 8031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Generated new TSK 0
I20260929 02:46:50.366245 8031 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260929 02:46:50.425030 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:50.754602 8044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:50.754899 8044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:50.755074 8044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:50.762604 8044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:50.762703 8044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:50.762840 8044 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:50.762970 8044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:50.772845 8044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:50.774225 8044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:50.775609 8044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:50.789202 8050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:50.789669 8052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:50.789703 8049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:50.790206 8044 server_base.cc:1034] running on GCE node
I20260929 02:46:50.790863 8044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:50.791817 8044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:50.793058 8044 hybrid_clock.cc:648] HybridClock initialized: now 1790650010793003 us; error 42 us; skew 500 ppm
I20260929 02:46:50.793378 8044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:50 dist-test-slave-6tjh krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650010, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:50.796672 8044 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 02:46:50.798187 8044 webserver.cc:460] Webserver started at http://127.0.97.1:42405/ using document root <none> and password file <none>
I20260929 02:46:50.798877 8044 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:50.799011 8044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:50.799328 8044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:50.802359 8044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "7d4027cb24ba4cb48bfa9178ce3270d9"
format_stamp: "Formatted at 2026-09-29 02:46:50 on dist-test-slave-6tjh"
server_key: "3aebfa63612f8f63be802d401842c9b6"
server_key_iv: "dc957e49bf83da1c2cae3acbc0f777e9"
server_key_version: "encryptionkey@0"
I20260929 02:46:50.803188 8044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "7d4027cb24ba4cb48bfa9178ce3270d9"
format_stamp: "Formatted at 2026-09-29 02:46:50 on dist-test-slave-6tjh"
server_key: "3aebfa63612f8f63be802d401842c9b6"
server_key_iv: "dc957e49bf83da1c2cae3acbc0f777e9"
server_key_version: "encryptionkey@0"
I20260929 02:46:50.808642 8044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:46:50.812155 8059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:50.813746 8044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:46:50.813932 8044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "7d4027cb24ba4cb48bfa9178ce3270d9"
format_stamp: "Formatted at 2026-09-29 02:46:50 on dist-test-slave-6tjh"
server_key: "3aebfa63612f8f63be802d401842c9b6"
server_key_iv: "dc957e49bf83da1c2cae3acbc0f777e9"
server_key_version: "encryptionkey@0"
I20260929 02:46:50.814178 8044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:50.846751 8044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:50.851152 8044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:50.851486 8044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:50.854544 8044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:50.854687 8044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:50.854841 8044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:50.854924 8044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:46:50 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650010, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:50.879352 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:50.855771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42573 (local address 127.0.97.62:36871)
0929 02:46:50.856258 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:50.856265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:50.857243 (+ 978us) server_negotiation.cc:407] Connection header received
0929 02:46:50.859864 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:50.859869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:50.859965 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:50.860091 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:50.862719 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.863776 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.866225 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.866805 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.866950 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:50.872103 (+ 5153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:50.872109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:50.872112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:50.872145 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:50.875737 (+ 3592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:50.876971 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:50.876975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:50.876978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:50.877125 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:50.877580 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:50.877587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:50.877595 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:50.877892 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:50.878047 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:50.878771 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 02:46:50.878989 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":243,"threads_started":1}
I20260929 02:46:50.880373 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:50.856123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.1:42573)
0929 02:46:50.857039 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:50.857087 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:50.859420 (+ 2333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:50.860554 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:50.860594 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:50.861485 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:50.862423 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:50.862462 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.863935 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.863946 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:50.866002 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:50.866015 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.867105 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.867112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:50.867464 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:50.868365 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:50.868412 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:50.871827 (+ 3415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:50.875918 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:50.875925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:50.875944 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:50.876729 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:50.877245 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:50.877251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:50.877255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:50.877455 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:50.878021 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:50.878028 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:50.878402 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 02:46:50.878669 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 02:46:50.879223 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":421,"threads_started":1}
I20260929 02:46:50.902565 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:50.887671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46313 (local address 127.0.97.62:36871)
0929 02:46:50.887909 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:50.887915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:50.888630 (+ 715us) server_negotiation.cc:407] Connection header received
0929 02:46:50.889045 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:50.889054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:50.889152 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:50.889299 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:50.890709 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.891434 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.892823 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.893415 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.893560 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:50.895948 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:50.895957 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:50.895963 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:50.896002 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:50.898990 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:50.900101 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:50.900110 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:50.900116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:50.900313 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:50.900802 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:50.900810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:50.900814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:50.901228 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:50.901412 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:50.902131 (+ 719us) server_negotiation.cc:299] Negotiation successful
0929 02:46:50.902323 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 02:46:50.903164 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:50.887704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.1:46313)
0929 02:46:50.888471 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:50.888495 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:50.888778 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:50.889343 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:50.889350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:50.890051 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:50.890563 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:50.890574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.891571 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.891577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:50.892676 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:50.892687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:50.893604 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:50.893609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:50.893782 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:50.894461 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:50.894491 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:50.895747 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:50.899140 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:50.899147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:50.899151 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:50.899949 (+ 798us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:50.900457 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:50.900465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:50.900469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:50.900671 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:50.901393 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:50.901406 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:50.901684 (+ 278us) client_negotiation.cc:769] Sending connection context
0929 02:46:50.902002 (+ 318us) client_negotiation.cc:240] Negotiation successful
0929 02:46:50.902550 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":488,"threads_started":1}
I20260929 02:46:50.913386 8044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35129
I20260929 02:46:50.913407 8179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35129 every 8 connection(s)
I20260929 02:46:50.914590 8044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:50.916597 8180 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36871
I20260929 02:46:50.916839 8180 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:50.917534 8180 heartbeater.cc:511] Master 127.0.97.62:36871 requested a full tablet report, sending...
I20260929 02:46:50.919445 7973 ts_manager.cc:194] Registered new tserver with Master: 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129)
I20260929 02:46:50.919996 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 8044
I20260929 02:46:50.920192 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:50.921015 388 external_mini_cluster.cc:1428] Setting key 10c1d0494b05a54994aa076a3268e39c
I20260929 02:46:50.921566 7973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42573
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260929 02:46:50.968302 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:51.304945 8184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:51.305245 8184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:51.305413 8184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:51.313515 8184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:51.313633 8184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:51.313813 8184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:51.313941 8184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:51.324195 8184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:51.325529 8184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:51.326870 8184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:51.341837 8192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:51.341943 8190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:51.341907 8189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:51.342959 8184 server_base.cc:1034] running on GCE node
I20260929 02:46:51.343735 8184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:51.344880 8184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:51.346189 8184 hybrid_clock.cc:648] HybridClock initialized: now 1790650011346115 us; error 57 us; skew 500 ppm
I20260929 02:46:51.346517 8184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:51 dist-test-slave-6tjh krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:51.350133 8184 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 02:46:51.351670 8184 webserver.cc:460] Webserver started at http://127.0.97.2:36279/ using document root <none> and password file <none>
I20260929 02:46:51.352413 8184 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:51.352586 8184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:51.352936 8184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:51.356273 8184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "4254ef0b0d4e4ea6a6f01d4011786c27"
format_stamp: "Formatted at 2026-09-29 02:46:51 on dist-test-slave-6tjh"
server_key: "5c8a9598d748584b427e62e90dd73dd7"
server_key_iv: "ba11c57fc6f10e112516f3fec9597891"
server_key_version: "encryptionkey@0"
I20260929 02:46:51.357390 8184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4254ef0b0d4e4ea6a6f01d4011786c27"
format_stamp: "Formatted at 2026-09-29 02:46:51 on dist-test-slave-6tjh"
server_key: "5c8a9598d748584b427e62e90dd73dd7"
server_key_iv: "ba11c57fc6f10e112516f3fec9597891"
server_key_version: "encryptionkey@0"
I20260929 02:46:51.363353 8184 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 02:46:51.367105 8199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:51.368703 8184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:46:51.368916 8184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "4254ef0b0d4e4ea6a6f01d4011786c27"
format_stamp: "Formatted at 2026-09-29 02:46:51 on dist-test-slave-6tjh"
server_key: "5c8a9598d748584b427e62e90dd73dd7"
server_key_iv: "ba11c57fc6f10e112516f3fec9597891"
server_key_version: "encryptionkey@0"
I20260929 02:46:51.369195 8184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:51.393437 8184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:51.398317 8184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:51.398623 8184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:51.401731 8184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:51.401862 8184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:51.402181 8184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:51.402282 8184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 29 02:46:51 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:51.426841 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.402084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60597 (local address 127.0.97.62:36871)
0929 02:46:51.402303 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:51.402309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:51.403877 (+ 1568us) server_negotiation.cc:407] Connection header received
0929 02:46:51.406005 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:51.406011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:51.406104 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:51.406227 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:51.408295 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.409878 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.412802 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.413436 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.413592 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:51.419093 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:51.419100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:51.419107 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:51.419157 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:51.422691 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:51.423807 (+ 1116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:51.423815 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:51.423822 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:51.424016 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:51.424474 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:51.424481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:51.424487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:51.425019 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:51.425205 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:51.425990 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 02:46:51.426329 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:46:51.427240 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.402769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.2:60597)
0929 02:46:51.403673 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:51.403727 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:51.405709 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:51.406484 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:51.406492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:51.407073 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:51.408040 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.408068 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.410098 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.410106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:51.412552 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.412602 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.413736 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.413746 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:51.413986 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:51.414825 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:51.414865 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:51.418782 (+ 3917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:51.422861 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:51.422874 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:51.422900 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:51.423617 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:51.424144 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:51.424148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:51.424150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:51.424346 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:51.425151 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:51.425157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:51.425531 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 02:46:51.425766 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 02:46:51.426161 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":347,"threads_started":1}
I20260929 02:46:51.450146 8068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.433419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60251 (local address 127.0.97.62:36871)
0929 02:46:51.433657 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:51.433664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:51.434286 (+ 622us) server_negotiation.cc:407] Connection header received
0929 02:46:51.434449 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:51.434458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:51.434566 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:51.434706 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:51.435637 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.437050 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.439150 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.439703 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.439868 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:51.443266 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:51.443275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:51.443283 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:51.443350 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:51.446777 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:51.447722 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:51.447732 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:51.447737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:51.447911 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:51.448415 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:51.448424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:51.448427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:51.448937 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:51.449105 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:51.449545 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 02:46:51.449775 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:46:51.450253 8272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.433471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.2:60251)
0929 02:46:51.434177 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:51.434193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:51.434315 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:51.434738 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:51.434741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:51.435218 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:51.435496 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.435504 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.437236 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.437243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:51.438903 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.438915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.439927 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.439938 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:51.440307 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:51.441329 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:51.441360 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:51.442930 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:51.446934 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:51.446941 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:51.446947 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:51.447554 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:51.448100 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:51.448107 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:51.448113 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:51.448283 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:51.449065 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:51.449070 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:51.449275 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 02:46:51.449454 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 02:46:51.449798 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":390,"threads_started":1}
I20260929 02:46:51.456488 8184 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40571
I20260929 02:46:51.456506 8318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40571 every 8 connection(s)
I20260929 02:46:51.457773 8184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:51.459944 8319 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36871
I20260929 02:46:51.460285 8319 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:51.460446 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 8184
I20260929 02:46:51.460650 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:51.461315 8319 heartbeater.cc:511] Master 127.0.97.62:36871 requested a full tablet report, sending...
I20260929 02:46:51.461390 388 external_mini_cluster.cc:1428] Setting key 76a0bfb2fd627261685448c327fd17fd
I20260929 02:46:51.462813 7973 ts_manager.cc:194] Registered new tserver with Master: 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571)
I20260929 02:46:51.463658 7973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:60597
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 02:46:51.517048 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36871
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:51.865057 8323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:51.865474 8323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:51.865664 8323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:51.873504 8323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:51.873679 8323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:51.873834 8323 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 02:46:51.873989 8323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:51.884400 8323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:36871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:51.885785 8323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:51.887243 8323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:51.900867 8331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:51.900943 8328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:51.900905 8329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:51.902136 8323 server_base.cc:1034] running on GCE node
I20260929 02:46:51.902770 8323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:51.903776 8323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:51.905014 8323 hybrid_clock.cc:648] HybridClock initialized: now 1790650011904944 us; error 44 us; skew 500 ppm
I20260929 02:46:51.905330 8323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:51 dist-test-slave-6tjh krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:51.908622 8323 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 02:46:51.910126 8323 webserver.cc:460] Webserver started at http://127.0.97.3:39717/ using document root <none> and password file <none>
I20260929 02:46:51.910825 8323 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:51.910936 8323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:51.911275 8323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:51.914160 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "8489bd3d3e7b4ab496b480c999e453a9"
format_stamp: "Formatted at 2026-09-29 02:46:51 on dist-test-slave-6tjh"
server_key: "4fd1c930bb90d2d8ba48662e11cf2eba"
server_key_iv: "a7f500fbeca71c22d247ead2f6847c0d"
server_key_version: "encryptionkey@0"
I20260929 02:46:51.914952 8323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8489bd3d3e7b4ab496b480c999e453a9"
format_stamp: "Formatted at 2026-09-29 02:46:51 on dist-test-slave-6tjh"
server_key: "4fd1c930bb90d2d8ba48662e11cf2eba"
server_key_iv: "a7f500fbeca71c22d247ead2f6847c0d"
server_key_version: "encryptionkey@0"
I20260929 02:46:51.920081 8323 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:46:51.923629 8338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:51.924445 8180 heartbeater.cc:503] Master 127.0.97.62:36871 was elected leader, sending a full tablet report...
I20260929 02:46:51.925275 8323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:46:51.925453 8323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "8489bd3d3e7b4ab496b480c999e453a9"
format_stamp: "Formatted at 2026-09-29 02:46:51 on dist-test-slave-6tjh"
server_key: "4fd1c930bb90d2d8ba48662e11cf2eba"
server_key_iv: "a7f500fbeca71c22d247ead2f6847c0d"
server_key_version: "encryptionkey@0"
I20260929 02:46:51.925678 8323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:51.946789 8323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:51.951468 8323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:51.951769 8323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:51.954452 8323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:51.954559 8323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:51.954711 8323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:51.954795 8323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:46:51 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650011, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:51.981407 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.955021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44431 (local address 127.0.97.62:36871)
0929 02:46:51.957860 (+ 2839us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:51.957865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:51.957884 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:46:51.960260 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:51.960267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:51.960367 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:51.960516 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:51.962725 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.963901 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.966634 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.967273 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.967416 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:51.973007 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:51.973017 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:51.973025 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:51.973075 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:51.976723 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:51.977805 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:51.977811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:51.977815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:51.978027 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:51.978721 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:51.978727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:51.978731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:51.979149 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:51.979338 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:51.980629 (+ 1291us) server_negotiation.cc:299] Negotiation successful
0929 02:46:51.980898 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2727,"thread_start_us":2636,"threads_started":1}
I20260929 02:46:51.983413 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.955850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.3:44431)
0929 02:46:51.956874 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:51.956931 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:51.959810 (+ 2879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:51.960806 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:51.960814 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:51.961568 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:51.962534 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.962564 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.964065 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.964071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:51.966474 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.966492 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.967572 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.967582 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:51.967923 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:51.968886 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:51.968928 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:51.972696 (+ 3768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:51.976883 (+ 4187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:51.976897 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:51.976920 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:51.977633 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:51.978191 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:51.978199 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:51.978206 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:51.978539 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:51.979418 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:51.979432 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:51.980041 (+ 609us) client_negotiation.cc:769] Sending connection context
0929 02:46:51.980467 (+ 426us) client_negotiation.cc:240] Negotiation successful
0929 02:46:51.981273 (+ 806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":394,"threads_started":1}
I20260929 02:46:51.999430 8323 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44549
I20260929 02:46:51.999452 8458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44549 every 8 connection(s)
I20260929 02:46:52.001034 8323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:52.003441 8459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36871
I20260929 02:46:52.003754 8459 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:52.004647 8459 heartbeater.cc:511] Master 127.0.97.62:36871 requested a full tablet report, sending...
I20260929 02:46:52.006403 7972 ts_manager.cc:194] Registered new tserver with Master: 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549)
I20260929 02:46:52.007472 7972 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44431
I20260929 02:46:52.009106 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.989898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40467 (local address 127.0.97.62:36871)
0929 02:46:51.990142 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:51.990150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:51.993725 (+ 3575us) server_negotiation.cc:407] Connection header received
0929 02:46:51.994059 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:51.994064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:51.994152 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:51.994279 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:51.996133 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.997454 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.999919 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.000309 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.000407 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.002917 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:52.002922 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:52.002926 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:52.002984 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:52.006347 (+ 3363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:52.007081 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:52.007086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:52.007090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:52.007239 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:52.007572 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:52.007576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:52.007579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:52.007920 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:52.008087 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.008475 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.008743 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:46:52.009230 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:51.989765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.97.3:40467)
0929 02:46:51.993600 (+ 3835us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:51.993624 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:51.993880 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:51.994375 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:51.994383 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:51.995281 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:51.995899 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.995914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:51.997650 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:51.997654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:51.999759 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:51.999775 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.000508 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.000516 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.000726 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.001403 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:52.001426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:52.002692 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:52.006525 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:52.006531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:52.006536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:52.006962 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:52.007351 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:52.007354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:52.007357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:52.007467 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:52.008042 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:52.008047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:52.008239 (+ 192us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.008390 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.008752 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3624,"thread_start_us":3382,"threads_started":1}
I20260929 02:46:52.010102 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 8323
I20260929 02:46:52.010270 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:52.010819 388 external_mini_cluster.cc:1428] Setting key 65fbe31a91baf8f290624c043be50490
I20260929 02:46:52.013756 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 02:46:52 dist-test-slave-6tjh krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 02:46:52 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:46:52 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:46:52.062417 8347 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:52.052042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.0.97.62:36871)
0929 02:46:52.052273 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.052280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.052456 (+ 176us) server_negotiation.cc:407] Connection header received
0929 02:46:52.052692 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.052696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.052777 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.052867 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:52.054307 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.055036 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.056408 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.056900 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.056997 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.062193 (+ 5196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59144: BlockingRecv error: recv got EOF from 127.0.0.1:59144 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 02:46:52 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:52.079056 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.065873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.97.62:36871)
0929 02:46:52.066101 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.066109 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.066300 (+ 191us) server_negotiation.cc:407] Connection header received
0929 02:46:52.066417 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.066420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.066480 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.066566 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:52.067638 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.068315 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.069582 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.070053 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.070167 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.073885 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:52.073890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:52.073893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:52.073927 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:52.076613 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:52.077276 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:52.077280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:52.077282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:52.077395 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:52.077820 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:52.077823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:52.077825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:52.078079 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:52.078268 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.078516 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.078674 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:46:52.083134 7972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:52.090956 7972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:52.139060 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.105766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.0.97.2:40571)
0929 02:46:52.121141 (+ 15375us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.121156 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.130212 (+ 9056us) server_negotiation.cc:407] Connection header received
0929 02:46:52.130748 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.130762 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.131081 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.131314 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.132213 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.133848 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.136780 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.137756 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.137946 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.138239 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.138596 (+ 357us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.138769 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15119,"thread_start_us":15389,"threads_started":1}
I20260929 02:46:52.139175 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.105567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40571 (local address 127.0.0.1:52502)
0929 02:46:52.129964 (+ 24397us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.130036 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.130428 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.131337 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.131344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.131379 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.131971 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.131987 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.133996 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.134001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.136525 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.136537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.138038 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.138044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.138374 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.138416 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.138557 (+ 141us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.138742 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24013,"mutex_wait_us":88,"thread_start_us":26235,"threads_started":1}
I20260929 02:46:52.139896 8479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.105935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44549 (local address 127.0.0.1:45656)
0929 02:46:52.131173 (+ 25238us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.131198 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.131453 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.132591 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.132596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.132622 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.133033 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.133046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.134758 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.134766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.136620 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.136647 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.139040 (+ 2393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.139050 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.139251 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.139288 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.139379 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.139510 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25078,"mutex_wait_us":58,"thread_start_us":24887,"threads_started":1}
I20260929 02:46:52.140317 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.106721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45656 (local address 127.0.97.3:44549)
0929 02:46:52.125307 (+ 18586us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.125318 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.131312 (+ 5994us) server_negotiation.cc:407] Connection header received
0929 02:46:52.131715 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.131724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.131988 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.132187 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.133305 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.134617 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.136850 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.138512 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.138635 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.138887 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.139873 (+ 986us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.140030 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18347,"thread_start_us":18130,"threads_started":1}
I20260929 02:46:52.141682 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.106905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.0.97.1:35129)
0929 02:46:52.129152 (+ 22247us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.129165 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.132197 (+ 3032us) server_negotiation.cc:407] Connection header received
0929 02:46:52.132481 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.132488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.132720 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.132883 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.133583 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.134971 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.138594 (+ 3623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.140113 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.140239 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.140463 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.141160 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.141344 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22031,"thread_start_us":21872,"threads_started":1}
I20260929 02:46:52.142424 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.106360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35129 (local address 127.0.0.1:53140)
0929 02:46:52.132096 (+ 25736us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.132112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.132288 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.133112 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.133115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.133137 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.133414 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.133421 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.135927 (+ 2506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.135932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.138388 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.138403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.140708 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.140716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.140923 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.140960 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.141951 (+ 991us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.142073 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25614,"mutex_wait_us":42,"thread_start_us":23056,"threads_started":1}
I20260929 02:46:52.143455 8249 tablet_service.cc:1467] Processing CreateTablet for tablet bd4643da3cc5417e8261be2c4a8a0d86 (DEFAULT_TABLE table=test-table [id=59c40ef2245d4e2daa0790bf800e1dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:52.143913 8389 tablet_service.cc:1467] Processing CreateTablet for tablet bd4643da3cc5417e8261be2c4a8a0d86 (DEFAULT_TABLE table=test-table [id=59c40ef2245d4e2daa0790bf800e1dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:52.144971 8249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4643da3cc5417e8261be2c4a8a0d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.145334 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4643da3cc5417e8261be2c4a8a0d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.145946 8110 tablet_service.cc:1467] Processing CreateTablet for tablet bd4643da3cc5417e8261be2c4a8a0d86 (DEFAULT_TABLE table=test-table [id=59c40ef2245d4e2daa0790bf800e1dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:52.147156 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4643da3cc5417e8261be2c4a8a0d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.157819 8483 tablet_bootstrap.cc:492] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Bootstrap starting.
I20260929 02:46:52.159842 8484 tablet_bootstrap.cc:492] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Bootstrap starting.
I20260929 02:46:52.161167 8485 tablet_bootstrap.cc:492] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Bootstrap starting.
I20260929 02:46:52.162089 8483 tablet_bootstrap.cc:654] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.163451 8484 tablet_bootstrap.cc:654] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.163901 8485 tablet_bootstrap.cc:654] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.164412 8483 log.cc:824] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:52.164832 8484 log.cc:824] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:52.165069 8485 log.cc:824] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:52.168769 8484 tablet_bootstrap.cc:492] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: No bootstrap required, opened a new log
I20260929 02:46:52.169129 8484 ts_tablet_manager.cc:1397] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 02:46:52.176206 8485 tablet_bootstrap.cc:492] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: No bootstrap required, opened a new log
I20260929 02:46:52.176205 8483 tablet_bootstrap.cc:492] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: No bootstrap required, opened a new log
I20260929 02:46:52.176512 8485 ts_tablet_manager.cc:1397] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260929 02:46:52.176512 8483 ts_tablet_manager.cc:1397] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20260929 02:46:52.179103 8484 raft_consensus.cc:348] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.179538 8484 raft_consensus.cc:374] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.179692 8484 raft_consensus.cc:729] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4254ef0b0d4e4ea6a6f01d4011786c27, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.180371 8484 consensus_queue.cc:260] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.181679 8319 heartbeater.cc:503] Master 127.0.97.62:36871 was elected leader, sending a full tablet report...
I20260929 02:46:52.182493 8484 ts_tablet_manager.cc:1428] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 02:46:52.187511 8485 raft_consensus.cc:348] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.187515 8483 raft_consensus.cc:348] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.187979 8485 raft_consensus.cc:374] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.187979 8483 raft_consensus.cc:374] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.188077 8483 raft_consensus.cc:729] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8489bd3d3e7b4ab496b480c999e453a9, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.188154 8485 raft_consensus.cc:729] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4027cb24ba4cb48bfa9178ce3270d9, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.188787 8483 consensus_queue.cc:260] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.188791 8485 consensus_queue.cc:260] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.190379 8459 heartbeater.cc:503] Master 127.0.97.62:36871 was elected leader, sending a full tablet report...
I20260929 02:46:52.190713 8485 ts_tablet_manager.cc:1428] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:46:52.190886 8483 ts_tablet_manager.cc:1428] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 02:46:52.191339 8489 raft_consensus.cc:482] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:52.191618 8489 raft_consensus.cc:504] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.193538 8489 leader_election.cc:288] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549), 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129)
I20260929 02:46:52.206869 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.194394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32835 (local address 127.0.97.1:35129)
0929 02:46:52.194658 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.194667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.201435 (+ 6768us) server_negotiation.cc:407] Connection header received
0929 02:46:52.201661 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.201665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.201725 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.201833 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.202496 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.203228 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.205153 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.205914 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.206075 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.206197 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.206478 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.206612 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260929 02:46:52.207016 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.193827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35129 (local address 127.0.97.2:32835)
0929 02:46:52.201291 (+ 7464us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.201320 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.201511 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.201874 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.201881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.201907 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.202353 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.202362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.203358 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.203365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.204995 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.205007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.206141 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.206147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.206316 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.206340 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.206452 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.206600 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7267,"thread_start_us":7980,"threads_started":1}
I20260929 02:46:52.207319 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.194230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59547 (local address 127.0.97.3:44549)
0929 02:46:52.194402 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.194409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.202272 (+ 7863us) server_negotiation.cc:407] Connection header received
0929 02:46:52.202455 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.202460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.202525 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.202617 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.203124 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.203805 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.205588 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.206373 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.206529 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.206636 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.206934 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.207062 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 02:46:52.207413 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.194000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44549 (local address 127.0.97.2:59547)
0929 02:46:52.202155 (+ 8155us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.202172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.202323 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.202640 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.202645 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.202668 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.202981 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.202989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.203957 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.203962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.205430 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.205441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.206590 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.206595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.206779 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.206801 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.206910 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.207053 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8030,"thread_start_us":6817,"threads_started":1}
I20260929 02:46:52.208169 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4643da3cc5417e8261be2c4a8a0d86" candidate_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" is_pre_election: true
I20260929 02:46:52.208360 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4643da3cc5417e8261be2c4a8a0d86" candidate_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8489bd3d3e7b4ab496b480c999e453a9" is_pre_election: true
I20260929 02:46:52.208775 8134 raft_consensus.cc:2454] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4254ef0b0d4e4ea6a6f01d4011786c27 in term 0.
I20260929 02:46:52.209000 8409 raft_consensus.cc:2454] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4254ef0b0d4e4ea6a6f01d4011786c27 in term 0.
I20260929 02:46:52.209548 8200 leader_election.cc:302] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4254ef0b0d4e4ea6a6f01d4011786c27, 7d4027cb24ba4cb48bfa9178ce3270d9; no voters:
W20260929 02:46:52.209697 8320 tablet.cc:2369] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:52.210029 8489 raft_consensus.cc:2781] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:52.210176 8489 raft_consensus.cc:482] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:52.210295 8489 raft_consensus.cc:3037] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.212419 8489 raft_consensus.cc:504] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.213174 8489 leader_election.cc:288] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [CANDIDATE]: Term 1 election: Requested vote from peers 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549), 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129)
I20260929 02:46:52.213560 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4643da3cc5417e8261be2c4a8a0d86" candidate_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8489bd3d3e7b4ab496b480c999e453a9"
I20260929 02:46:52.213670 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4643da3cc5417e8261be2c4a8a0d86" candidate_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9"
I20260929 02:46:52.213776 8409 raft_consensus.cc:3037] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.213888 8134 raft_consensus.cc:3037] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.216037 8409 raft_consensus.cc:2454] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4254ef0b0d4e4ea6a6f01d4011786c27 in term 1.
I20260929 02:46:52.216084 8134 raft_consensus.cc:2454] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4254ef0b0d4e4ea6a6f01d4011786c27 in term 1.
I20260929 02:46:52.216495 8202 leader_election.cc:302] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4254ef0b0d4e4ea6a6f01d4011786c27, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:52.216892 8489 raft_consensus.cc:2781] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:52.217325 8489 raft_consensus.cc:686] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 LEADER]: Becoming Leader. State: Replica: 4254ef0b0d4e4ea6a6f01d4011786c27, State: Running, Role: LEADER
I20260929 02:46:52.217918 8489 consensus_queue.cc:237] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.222661 7972 catalog_manager.cc:5495] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:52.244513 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.239216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.0.97.2:40571)
0929 02:46:52.239415 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.239419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.239436 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:46:52.239615 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.239619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.239702 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.239826 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:52.240110 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.240928 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.242300 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.242778 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.242892 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.243298 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:52.243859 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:52.243968 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.244037 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.244100 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 02:46:52.252882 8460 tablet.cc:2369] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:52.253108 8134 raft_consensus.cc:1261] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Refusing update from remote peer 4254ef0b0d4e4ea6a6f01d4011786c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:52.253136 8409 raft_consensus.cc:1261] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Refusing update from remote peer 4254ef0b0d4e4ea6a6f01d4011786c27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:52.254103 8494 consensus_queue.cc:1035] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:52.254549 8489 consensus_queue.cc:1035] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:52.286105 7972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:52.286481 7972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:52.290606 7972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Sending DeleteTablet for 3 replicas of tablet bd4643da3cc5417e8261be2c4a8a0d86
I20260929 02:46:52.291873 8249 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4643da3cc5417e8261be2c4a8a0d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52502
I20260929 02:46:52.291893 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4643da3cc5417e8261be2c4a8a0d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53140
I20260929 02:46:52.292618 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4643da3cc5417e8261be2c4a8a0d86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:52 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45656
I20260929 02:46:52.293687 7972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:52.294827 7972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:52.305022 8388 tablet_service.cc:1467] Processing CreateTablet for tablet 29b9b6d3e81445afba3193c5a11a710a (DEFAULT_TABLE table=test-table [id=131f1bf026804e8fa66945aba74e8b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:52.305390 8109 tablet_service.cc:1467] Processing CreateTablet for tablet 29b9b6d3e81445afba3193c5a11a710a (DEFAULT_TABLE table=test-table [id=131f1bf026804e8fa66945aba74e8b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:52.305716 8388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b9b6d3e81445afba3193c5a11a710a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.306115 8109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b9b6d3e81445afba3193c5a11a710a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.314000 8483 tablet_bootstrap.cc:492] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: Bootstrap starting.
I20260929 02:46:52.315199 8509 tablet_replica.cc:330] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: stopping tablet replica
I20260929 02:46:52.315825 8483 tablet_bootstrap.cc:654] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.315748 8509 raft_consensus.cc:2229] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:52.316794 8509 raft_consensus.cc:2258] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:52.319571 8509 ts_tablet_manager.cc:1905] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:52.322103 8507 tablet_replica.cc:330] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: stopping tablet replica
I20260929 02:46:52.321933 8248 tablet_service.cc:1467] Processing CreateTablet for tablet 29b9b6d3e81445afba3193c5a11a710a (DEFAULT_TABLE table=test-table [id=131f1bf026804e8fa66945aba74e8b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:52.322786 8248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b9b6d3e81445afba3193c5a11a710a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.325947 8485 tablet_bootstrap.cc:492] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Bootstrap starting.
I20260929 02:46:52.328259 8485 tablet_bootstrap.cc:654] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.333222 8507 raft_consensus.cc:2229] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:52.334152 8507 raft_consensus.cc:2258] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:52.335623 8485 tablet_bootstrap.cc:492] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: No bootstrap required, opened a new log
I20260929 02:46:52.335850 8485 ts_tablet_manager.cc:1397] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260929 02:46:52.337443 8485 raft_consensus.cc:348] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.337818 8485 raft_consensus.cc:374] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.337404 8484 tablet_bootstrap.cc:492] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: Bootstrap starting.
I20260929 02:46:52.337993 8485 raft_consensus.cc:729] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4027cb24ba4cb48bfa9178ce3270d9, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.338474 8485 consensus_queue.cc:260] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.340019 8484 tablet_bootstrap.cc:654] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.340225 8485 ts_tablet_manager.cc:1428] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 02:46:52.346793 8507 ts_tablet_manager.cc:1905] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:52.352602 8508 tablet_replica.cc:330] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: stopping tablet replica
I20260929 02:46:52.353237 8508 raft_consensus.cc:2229] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:52.353888 8508 raft_consensus.cc:2258] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:52.356362 8508 ts_tablet_manager.cc:1905] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:52.357532 8483 tablet_bootstrap.cc:492] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: No bootstrap required, opened a new log
I20260929 02:46:52.357741 8483 ts_tablet_manager.cc:1397] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.020s
I20260929 02:46:52.358610 8483 raft_consensus.cc:348] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.358925 8483 raft_consensus.cc:374] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.359061 8483 raft_consensus.cc:729] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8489bd3d3e7b4ab496b480c999e453a9, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.359419 8483 consensus_queue.cc:260] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.360867 8483 ts_tablet_manager.cc:1428] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:46:52.363596 8508 ts_tablet_manager.cc:1918] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:52.363862 8508 log.cc:1196] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/bd4643da3cc5417e8261be2c4a8a0d86
I20260929 02:46:52.364511 8508 ts_tablet_manager.cc:1939] T bd4643da3cc5417e8261be2c4a8a0d86 P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting consensus metadata
I20260929 02:46:52.365602 8484 tablet_bootstrap.cc:492] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: No bootstrap required, opened a new log
I20260929 02:46:52.365792 8484 ts_tablet_manager.cc:1397] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.008s
I20260929 02:46:52.366652 7957 catalog_manager.cc:4841] TS 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129): tablet bd4643da3cc5417e8261be2c4a8a0d86 (table test-table [id=59c40ef2245d4e2daa0790bf800e1dc8]) successfully deleted
I20260929 02:46:52.366653 8484 raft_consensus.cc:348] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.366901 8484 raft_consensus.cc:374] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.367041 8484 raft_consensus.cc:729] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4254ef0b0d4e4ea6a6f01d4011786c27, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.367316 8484 consensus_queue.cc:260] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.367836 8484 ts_tablet_manager.cc:1428] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:46:52.373273 8509 ts_tablet_manager.cc:1918] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:52.373472 8509 log.cc:1196] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/bd4643da3cc5417e8261be2c4a8a0d86
I20260929 02:46:52.374060 8509 ts_tablet_manager.cc:1939] T bd4643da3cc5417e8261be2c4a8a0d86 P 8489bd3d3e7b4ab496b480c999e453a9: Deleting consensus metadata
I20260929 02:46:52.374210 8507 ts_tablet_manager.cc:1918] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:52.374383 8507 log.cc:1196] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/bd4643da3cc5417e8261be2c4a8a0d86
I20260929 02:46:52.374965 8507 ts_tablet_manager.cc:1939] T bd4643da3cc5417e8261be2c4a8a0d86 P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting consensus metadata
I20260929 02:46:52.376111 7959 catalog_manager.cc:4841] TS 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549): tablet bd4643da3cc5417e8261be2c4a8a0d86 (table test-table [id=59c40ef2245d4e2daa0790bf800e1dc8]) successfully deleted
I20260929 02:46:52.376516 7960 catalog_manager.cc:4841] TS 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571): tablet bd4643da3cc5417e8261be2c4a8a0d86 (table test-table [id=59c40ef2245d4e2daa0790bf800e1dc8]) successfully deleted
W20260929 02:46:52.417181 8181 tablet.cc:2369] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:46:52.447497 8491 raft_consensus.cc:482] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:52.447786 8491 raft_consensus.cc:504] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.449759 8491 leader_election.cc:288] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129), 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571)
I20260929 02:46:52.457137 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.450046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40571 (local address 127.0.97.3:57889)
0929 02:46:52.450317 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.450337 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.450500 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.451133 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.451139 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.451164 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.451353 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.451365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.452585 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.452590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.454392 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.454407 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.456259 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.456265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.456478 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.456513 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.456650 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.456846 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1}
I20260929 02:46:52.457312 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.450289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57889 (local address 127.0.97.2:40571)
0929 02:46:52.450772 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.450781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.450800 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:46:52.450872 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.450877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.451002 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.451102 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.451530 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.452415 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.454558 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.455908 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.456065 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.456171 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.456866 (+ 695us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.457017 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260929 02:46:52.458271 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.450857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51037 (local address 127.0.97.1:35129)
0929 02:46:52.451033 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.451041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.451062 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:46:52.451129 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.451133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.451227 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.451342 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.451938 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.453045 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.455124 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.456157 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.456276 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.456379 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.457841 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.457980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:46:52.458506 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b9b6d3e81445afba3193c5a11a710a" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" is_pre_election: true
I20260929 02:46:52.458932 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.450078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35129 (local address 127.0.97.3:51037)
0929 02:46:52.450797 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.450816 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.450947 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.451360 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.451365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.451390 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.451772 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.451786 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.453201 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.453207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.454978 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.454987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.456634 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.456642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.457639 (+ 997us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.457675 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.458434 (+ 759us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.458583 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":785,"thread_start_us":389,"threads_started":1}
I20260929 02:46:52.458972 8274 raft_consensus.cc:2454] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 0.
I20260929 02:46:52.459592 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b9b6d3e81445afba3193c5a11a710a" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" is_pre_election: true
I20260929 02:46:52.459822 8134 raft_consensus.cc:2454] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 0.
I20260929 02:46:52.460035 8342 leader_election.cc:302] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4254ef0b0d4e4ea6a6f01d4011786c27, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:52.460461 8491 raft_consensus.cc:2781] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:52.460603 8491 raft_consensus.cc:482] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:52.460705 8491 raft_consensus.cc:3037] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.463866 8491 raft_consensus.cc:504] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.464956 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b9b6d3e81445afba3193c5a11a710a" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9"
I20260929 02:46:52.465025 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b9b6d3e81445afba3193c5a11a710a" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27"
I20260929 02:46:52.465163 8134 raft_consensus.cc:3037] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.465248 8274 raft_consensus.cc:3037] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.468159 8134 raft_consensus.cc:2454] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 1.
I20260929 02:46:52.468736 8274 raft_consensus.cc:2454] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 1.
I20260929 02:46:52.468777 8339 leader_election.cc:302] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4027cb24ba4cb48bfa9178ce3270d9, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:52.469180 8491 leader_election.cc:288] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 election: Requested vote from peers 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129), 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571)
I20260929 02:46:52.469401 8491 raft_consensus.cc:2781] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:52.469532 8491 raft_consensus.cc:686] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 LEADER]: Becoming Leader. State: Replica: 8489bd3d3e7b4ab496b480c999e453a9, State: Running, Role: LEADER
I20260929 02:46:52.469935 8491 consensus_queue.cc:237] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:52.481194 7973 catalog_manager.cc:5495] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8489bd3d3e7b4ab496b480c999e453a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } health_report { overall_health: HEALTHY } } }
Sep 29 02:46:52 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650010, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:52.552645 8347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.539113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.0.97.62:36871)
0929 02:46:52.539373 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.539381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.540010 (+ 629us) server_negotiation.cc:407] Connection header received
0929 02:46:52.540188 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.540193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.540251 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.540352 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:52.541544 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.542283 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.543512 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.543914 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.544048 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.547410 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:52.547415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:52.547418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:52.547446 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:52.550030 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:52.550683 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:52.550689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:52.550696 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:52.550813 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:52.551154 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:52.551161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:52.551164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:52.551422 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:52.551565 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.551947 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.552182 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":119}
I20260929 02:46:52.552757 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.539125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36871 (local address 127.0.0.1:59170)
0929 02:46:52.539844 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.539893 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.540073 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.540372 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.540376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.541128 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:52.541400 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.541411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.542408 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.542416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.543371 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.543383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.544085 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.544091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.544263 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.545008 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:52.545030 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:52.547194 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:52.550153 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:52.550166 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:52.550173 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:52.550565 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:52.550913 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:52.550920 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:52.550923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:52.551052 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:52.551540 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:52.551548 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:52.551716 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.551876 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.552301 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":155,"thread_start_us":422,"threads_started":1}
I20260929 02:46:52.558065 7973 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59170:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 02:46:52.559101 7973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:52.565939 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 24d21417a4f54a05880eeff6c502a840 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05b2fa8c2bc04c6b92245b20b34a1297]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:52.565964 8248 tablet_service.cc:1467] Processing CreateTablet for tablet 24d21417a4f54a05880eeff6c502a840 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05b2fa8c2bc04c6b92245b20b34a1297]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:52.566474 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d21417a4f54a05880eeff6c502a840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.566493 8248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d21417a4f54a05880eeff6c502a840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.569454 8110 tablet_service.cc:1467] Processing CreateTablet for tablet 24d21417a4f54a05880eeff6c502a840 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=05b2fa8c2bc04c6b92245b20b34a1297]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 02:46:52.569988 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24d21417a4f54a05880eeff6c502a840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:52.572597 8483 tablet_bootstrap.cc:492] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: Bootstrap starting.
I20260929 02:46:52.574321 8483 tablet_bootstrap.cc:654] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.575464 8484 tablet_bootstrap.cc:492] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: Bootstrap starting.
I20260929 02:46:52.577517 8485 tablet_bootstrap.cc:492] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: Bootstrap starting.
I20260929 02:46:52.577658 8484 tablet_bootstrap.cc:654] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.579087 8485 tablet_bootstrap.cc:654] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:52.584384 8485 tablet_bootstrap.cc:492] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: No bootstrap required, opened a new log
I20260929 02:46:52.584460 8484 tablet_bootstrap.cc:492] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: No bootstrap required, opened a new log
I20260929 02:46:52.584614 8485 ts_tablet_manager.cc:1397] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260929 02:46:52.584656 8484 ts_tablet_manager.cc:1397] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 02:46:52.584906 8483 tablet_bootstrap.cc:492] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: No bootstrap required, opened a new log
I20260929 02:46:52.585083 8483 ts_tablet_manager.cc:1397] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260929 02:46:52.585379 8485 raft_consensus.cc:348] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.585471 8484 raft_consensus.cc:348] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.585618 8485 raft_consensus.cc:374] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.585683 8485 raft_consensus.cc:729] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4027cb24ba4cb48bfa9178ce3270d9, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.585706 8484 raft_consensus.cc:374] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.585799 8484 raft_consensus.cc:729] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4254ef0b0d4e4ea6a6f01d4011786c27, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.585840 8483 raft_consensus.cc:348] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.586155 8483 raft_consensus.cc:374] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:52.586012 8485 consensus_queue.cc:260] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.586058 8484 consensus_queue.cc:260] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.586227 8483 raft_consensus.cc:729] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8489bd3d3e7b4ab496b480c999e453a9, State: Initialized, Role: FOLLOWER
I20260929 02:46:52.586477 8490 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.586714 8484 ts_tablet_manager.cc:1428] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:46:52.586524 8504 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.586807 8490 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:52.586879 8504 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:52.587200 8483 consensus_queue.cc:260] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.587670 8491 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.587949 8491 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:52.588476 8483 ts_tablet_manager.cc:1428] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:52.590615 8485 ts_tablet_manager.cc:1428] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20260929 02:46:52.877491 8491 consensus_queue.cc:1035] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:52.881945 8490 raft_consensus.cc:482] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:52.882138 8490 raft_consensus.cc:504] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.883064 8491 consensus_queue.cc:1035] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:52.883965 8490 leader_election.cc:288] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571), 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549)
I20260929 02:46:52.905783 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.884429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44549 (local address 127.0.97.1:50549)
0929 02:46:52.898911 (+ 14482us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.898931 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.899116 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.899688 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.899692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.899723 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.900074 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.900088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.901718 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.901723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.903722 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.903734 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.904967 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.904972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.905152 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.905176 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.905299 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.905434 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14347,"thread_start_us":15915,"threads_started":1}
I20260929 02:46:52.905784 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.884597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50549 (local address 127.0.97.3:44549)
0929 02:46:52.898528 (+ 13931us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.898535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.899006 (+ 471us) server_negotiation.cc:407] Connection header received
0929 02:46:52.899276 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.899282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.899389 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.899467 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.900279 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.901530 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.903892 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.904776 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.904927 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.905030 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.905327 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.905459 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13804,"thread_start_us":13845,"threads_started":1}
I20260929 02:46:52.906670 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.885834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57059 (local address 127.0.97.2:40571)
0929 02:46:52.886064 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.886073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.900710 (+ 14637us) server_negotiation.cc:407] Connection header received
0929 02:46:52.900858 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.900863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.900959 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.901055 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:46:52.901658 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.902609 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.904815 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.905662 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.905835 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.905941 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.906274 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.906409 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66,"spinlock_wait_cycles":29440}
I20260929 02:46:52.906702 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.885321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40571 (local address 127.0.97.1:57059)
0929 02:46:52.900605 (+ 15284us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.900623 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.900731 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.901095 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.901100 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.901122 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:46:52.901465 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.901476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.902807 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.902812 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.904640 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.904655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.905916 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.905923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.906103 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.906139 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.906250 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.906377 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15129,"thread_start_us":13323,"threads_started":1}
I20260929 02:46:52.906790 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d21417a4f54a05880eeff6c502a840" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8489bd3d3e7b4ab496b480c999e453a9" is_pre_election: true
I20260929 02:46:52.907078 8409 raft_consensus.cc:2454] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 0.
I20260929 02:46:52.907572 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d21417a4f54a05880eeff6c502a840" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" is_pre_election: true
I20260929 02:46:52.907555 8062 leader_election.cc:302] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4027cb24ba4cb48bfa9178ce3270d9, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:52.907804 8274 raft_consensus.cc:2454] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 0.
I20260929 02:46:52.908030 8529 raft_consensus.cc:2781] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:52.908193 8529 raft_consensus.cc:482] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:52.908362 8529 raft_consensus.cc:3037] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.910607 8529 raft_consensus.cc:504] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.911276 8529 leader_election.cc:288] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571), 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549)
I20260929 02:46:52.911710 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d21417a4f54a05880eeff6c502a840" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27"
I20260929 02:46:52.911796 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24d21417a4f54a05880eeff6c502a840" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8489bd3d3e7b4ab496b480c999e453a9"
I20260929 02:46:52.911923 8274 raft_consensus.cc:3037] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.912053 8409 raft_consensus.cc:3037] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:52.914203 8409 raft_consensus.cc:2454] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 1.
I20260929 02:46:52.914628 8062 leader_election.cc:302] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4027cb24ba4cb48bfa9178ce3270d9, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:52.914659 8274 raft_consensus.cc:2454] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 1.
I20260929 02:46:52.914889 8529 raft_consensus.cc:2781] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:52.915076 8529 raft_consensus.cc:686] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4027cb24ba4cb48bfa9178ce3270d9, State: Running, Role: LEADER
I20260929 02:46:52.915479 8529 consensus_queue.cc:237] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } }
I20260929 02:46:52.917469 8490 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: TxnStatusTablet state changed. Reason: New leader 7d4027cb24ba4cb48bfa9178ce3270d9. Latest consensus state: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.917788 8490 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:52.922402 8536 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 02:46:52.922613 8536 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 02:46:52.922636 7973 catalog_manager.cc:5495] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } health_report { overall_health: HEALTHY } } }
I20260929 02:46:52.960894 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.955223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.0.97.1:35129)
0929 02:46:52.955408 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.955416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.955439 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:46:52.955725 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.955731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.955854 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.956001 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:52.956352 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.957180 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.958519 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.958960 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.959106 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.959576 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:52.960163 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:52.960340 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.960471 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.960586 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 02:46:52.960908 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.955119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35129 (local address 127.0.0.1:53146)
0929 02:46:52.955363 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:52.955383 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:52.955524 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:52.956019 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:52.956023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:52.956034 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:52.956211 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.956220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.957325 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.957332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.958378 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:52.958389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.959125 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.959131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:52.959323 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.959377 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:52.960286 (+ 909us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:52.960306 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 02:46:52.960440 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:46:52.960619 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 02:46:52.965487 8274 raft_consensus.cc:1261] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Refusing update from remote peer 7d4027cb24ba4cb48bfa9178ce3270d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:52.965478 8409 raft_consensus.cc:1261] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Refusing update from remote peer 7d4027cb24ba4cb48bfa9178ce3270d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:52.966215 8490 consensus_queue.cc:1035] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:52.966853 8529 consensus_queue.cc:1035] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:52.969563 8504 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: TxnStatusTablet state changed. Reason: New leader 7d4027cb24ba4cb48bfa9178ce3270d9. Latest consensus state: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.969941 8504 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:52.972059 8527 mvcc.cc:205] Tried to move back new op lower bound from 7334502453099401216 to 7334502452905574400. Current Snapshot: MvccSnapshot[applied={T|T < 7334502453099401216}]
I20260929 02:46:52.972759 8534 mvcc.cc:205] Tried to move back new op lower bound from 7334502453099401216 to 7334502452905574400. Current Snapshot: MvccSnapshot[applied={T|T < 7334502453099401216}]
I20260929 02:46:52.974386 8491 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: TxnStatusTablet state changed. Reason: New leader 7d4027cb24ba4cb48bfa9178ce3270d9. Latest consensus state: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.974779 8491 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:52.975502 8504 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.975716 8504 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 4254ef0b0d4e4ea6a6f01d4011786c27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:52.978600 8491 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.978857 8491 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 8489bd3d3e7b4ab496b480c999e453a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 02:46:52.985337 8529 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.985671 8529 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:52.986245 8543 tablet_replica.cc:439] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } }
I20260929 02:46:52.988247 8543 tablet_replica.cc:442] T 24d21417a4f54a05880eeff6c502a840 P 7d4027cb24ba4cb48bfa9178ce3270d9: This TxnStatusTablet replica's current role is: LEADER
I20260929 02:46:52.997002 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:52.990721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.0.97.3:44549)
0929 02:46:52.990992 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:52.990999 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:52.991225 (+ 226us) server_negotiation.cc:407] Connection header received
0929 02:46:52.991431 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:52.991435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:52.991518 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:52.991638 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:52.992100 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.992894 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.994799 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:52.995203 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:52.995309 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:52.995782 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:52.996377 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:52.996481 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:52.996635 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 02:46:52.996708 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:46:53.014098 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.006491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49589 (local address 127.0.97.1:35129)
0929 02:46:53.006712 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.006721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.009186 (+ 2465us) server_negotiation.cc:407] Connection header received
0929 02:46:53.009414 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.009418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.009481 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.009578 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:53.010071 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.010790 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.012193 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.012748 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.012894 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.013333 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:53.013435 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:53.013530 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:53.013689 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 02:46:53.013765 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:46:53.014176 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.006493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35129 (local address 127.0.97.3:49589)
0929 02:46:53.009079 (+ 2586us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:53.009095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:53.009250 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:53.009590 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:53.009595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:53.009606 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:53.009918 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:53.009926 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.010923 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.010931 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:53.012049 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:53.012061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.012940 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.012949 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:53.013094 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.013149 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:53.013529 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:53.013542 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 02:46:53.013642 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:46:53.013778 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2412,"thread_start_us":2285,"threads_started":1}
I20260929 02:46:53.053912 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.046691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42067 (local address 127.0.97.3:44549)
0929 02:46:53.046884 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.046894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.047628 (+ 734us) server_negotiation.cc:407] Connection header received
0929 02:46:53.047897 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.047902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.048009 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.048077 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:53.049055 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.049849 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.051593 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.052219 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.052327 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.053159 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:53.053274 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:53.053396 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:53.053508 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:46:53.053586 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:46:53.054150 8551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.046650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44549 (local address 127.0.97.1:42067)
0929 02:46:53.047525 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:53.047546 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:53.047745 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:53.048431 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:53.048435 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:53.048448 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 02:46:53.048864 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:53.048880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.049989 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.049992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:53.051411 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:53.051423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.052537 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.052543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:53.052728 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.052797 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 02:46:53.053399 (+ 602us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 02:46:53.053413 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 02:46:53.053544 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 02:46:53.053668 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":420,"threads_started":1}
I20260929 02:46:53.105412 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:53.105671 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:53.109194 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Sending DeleteTablet for 3 replicas of tablet 29b9b6d3e81445afba3193c5a11a710a
I20260929 02:46:53.110227 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b9b6d3e81445afba3193c5a11a710a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45656
I20260929 02:46:53.110611 8248 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b9b6d3e81445afba3193c5a11a710a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52502
I20260929 02:46:53.119271 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b9b6d3e81445afba3193c5a11a710a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53140
I20260929 02:46:53.120385 8555 tablet_replica.cc:330] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: stopping tablet replica
I20260929 02:46:53.121100 8555 raft_consensus.cc:2229] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:53.121683 8555 raft_consensus.cc:2258] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.123342 8555 ts_tablet_manager.cc:1905] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.129698 8556 tablet_replica.cc:330] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: stopping tablet replica
I20260929 02:46:53.129927 8556 raft_consensus.cc:2229] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:53.131599 8556 raft_consensus.cc:2258] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.132858 8556 ts_tablet_manager.cc:1905] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.136518 8555 ts_tablet_manager.cc:1918] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:53.140179 8555 log.cc:1196] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/29b9b6d3e81445afba3193c5a11a710a
I20260929 02:46:53.140204 8556 ts_tablet_manager.cc:1918] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:53.140357 8556 log.cc:1196] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/29b9b6d3e81445afba3193c5a11a710a
I20260929 02:46:53.140841 8556 ts_tablet_manager.cc:1939] T 29b9b6d3e81445afba3193c5a11a710a P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting consensus metadata
I20260929 02:46:53.141507 8555 ts_tablet_manager.cc:1939] T 29b9b6d3e81445afba3193c5a11a710a P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting consensus metadata
I20260929 02:46:53.142058 7957 catalog_manager.cc:4841] TS 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129): tablet 29b9b6d3e81445afba3193c5a11a710a (table test-table [id=131f1bf026804e8fa66945aba74e8b91]) successfully deleted
I20260929 02:46:53.142668 8554 tablet_replica.cc:330] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: stopping tablet replica
I20260929 02:46:53.142870 8554 raft_consensus.cc:2229] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:53.143373 7960 catalog_manager.cc:4841] TS 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571): tablet 29b9b6d3e81445afba3193c5a11a710a (table test-table [id=131f1bf026804e8fa66945aba74e8b91]) successfully deleted
I20260929 02:46:53.143450 8554 raft_consensus.cc:2258] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.144508 8554 ts_tablet_manager.cc:1905] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.152244 8554 ts_tablet_manager.cc:1918] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:53.152571 8554 log.cc:1196] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/29b9b6d3e81445afba3193c5a11a710a
I20260929 02:46:53.153088 8554 ts_tablet_manager.cc:1939] T 29b9b6d3e81445afba3193c5a11a710a P 8489bd3d3e7b4ab496b480c999e453a9: Deleting consensus metadata
I20260929 02:46:53.154330 7959 catalog_manager.cc:4841] TS 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549): tablet 29b9b6d3e81445afba3193c5a11a710a (table test-table [id=131f1bf026804e8fa66945aba74e8b91]) successfully deleted
Sep 29 02:46:53 dist-test-slave-6tjh krb5kdc[7928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:46:53 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:46:53 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:46:53.224027 8564 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:46:53.183566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.97.62:36871)
0929 02:46:53.196005 (+ 12439us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.196110 (+ 105us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.196918 (+ 808us) server_negotiation.cc:407] Connection header received
0929 02:46:53.196990 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.196995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.197079 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.197249 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:53.198883 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.200090 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.202027 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.202714 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.202856 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.215822 (+ 12966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59186: BlockingRecv error: recv got EOF from 127.0.0.1:59186 (error 108)
Metrics: {"server-negotiator.queue_time_us":12288,"thread_start_us":12456,"threads_started":1}
Sep 29 02:46:53 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260929 02:46:53.248260 8564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.227497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.0.97.62:36871)
0929 02:46:53.227678 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.227684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.229466 (+ 1782us) server_negotiation.cc:407] Connection header received
0929 02:46:53.229797 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.229804 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.229885 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.229961 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:53.231852 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.232987 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.234885 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.235555 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.235667 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.241029 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:53.241037 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:53.241042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:53.241095 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:53.244805 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:53.245747 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:53.245754 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:53.245759 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:53.245946 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:53.246444 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:53.246450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:53.246453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:53.246836 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:53.247056 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:53.247645 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 02:46:53.247877 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:46:53.253396 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:53.254137 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:53.262087 8248 tablet_service.cc:1467] Processing CreateTablet for tablet 8acc5f1158b441a791daf16b111a17ca (DEFAULT_TABLE table=test-table [id=0edf82eb67154475b16b70d9f7712cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:53.262418 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 8acc5f1158b441a791daf16b111a17ca (DEFAULT_TABLE table=test-table [id=0edf82eb67154475b16b70d9f7712cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:53.262470 8110 tablet_service.cc:1467] Processing CreateTablet for tablet 8acc5f1158b441a791daf16b111a17ca (DEFAULT_TABLE table=test-table [id=0edf82eb67154475b16b70d9f7712cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:53.262584 8248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8acc5f1158b441a791daf16b111a17ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:53.262799 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8acc5f1158b441a791daf16b111a17ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:53.262904 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8acc5f1158b441a791daf16b111a17ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:53.277248 8571 tablet_bootstrap.cc:492] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: Bootstrap starting.
I20260929 02:46:53.278942 8572 tablet_bootstrap.cc:492] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: Bootstrap starting.
I20260929 02:46:53.280933 8572 tablet_bootstrap.cc:654] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:53.280982 8571 tablet_bootstrap.cc:654] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:53.288275 8570 tablet_bootstrap.cc:492] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: Bootstrap starting.
I20260929 02:46:53.289381 8572 tablet_bootstrap.cc:492] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: No bootstrap required, opened a new log
I20260929 02:46:53.289559 8572 ts_tablet_manager.cc:1397] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260929 02:46:53.289894 8571 tablet_bootstrap.cc:492] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: No bootstrap required, opened a new log
I20260929 02:46:53.290055 8571 ts_tablet_manager.cc:1397] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260929 02:46:53.290334 8572 raft_consensus.cc:348] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.290609 8572 raft_consensus.cc:374] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:53.290710 8572 raft_consensus.cc:729] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4254ef0b0d4e4ea6a6f01d4011786c27, State: Initialized, Role: FOLLOWER
I20260929 02:46:53.290911 8571 raft_consensus.cc:348] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.291175 8571 raft_consensus.cc:374] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:53.292448 8571 raft_consensus.cc:729] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4027cb24ba4cb48bfa9178ce3270d9, State: Initialized, Role: FOLLOWER
I20260929 02:46:53.292733 8571 consensus_queue.cc:260] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.294298 8572 consensus_queue.cc:260] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.295670 8572 ts_tablet_manager.cc:1428] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 02:46:53.299355 8570 tablet_bootstrap.cc:654] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:53.301105 8571 ts_tablet_manager.cc:1428] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260929 02:46:53.305804 8570 tablet_bootstrap.cc:492] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: No bootstrap required, opened a new log
I20260929 02:46:53.306015 8570 ts_tablet_manager.cc:1397] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260929 02:46:53.312885 8570 raft_consensus.cc:348] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.313220 8570 raft_consensus.cc:374] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:53.313326 8570 raft_consensus.cc:729] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8489bd3d3e7b4ab496b480c999e453a9, State: Initialized, Role: FOLLOWER
I20260929 02:46:53.313612 8570 consensus_queue.cc:260] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.314313 8570 ts_tablet_manager.cc:1428] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260929 02:46:53.473920 8529 raft_consensus.cc:482] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:53.474166 8529 raft_consensus.cc:504] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.474854 8529 leader_election.cc:288] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571), 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549)
I20260929 02:46:53.475385 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acc5f1158b441a791daf16b111a17ca" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" is_pre_election: true
I20260929 02:46:53.475662 8274 raft_consensus.cc:2454] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 0.
I20260929 02:46:53.475521 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acc5f1158b441a791daf16b111a17ca" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8489bd3d3e7b4ab496b480c999e453a9" is_pre_election: true
I20260929 02:46:53.475893 8408 raft_consensus.cc:2454] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 0.
I20260929 02:46:53.476203 8063 leader_election.cc:302] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4254ef0b0d4e4ea6a6f01d4011786c27, 7d4027cb24ba4cb48bfa9178ce3270d9; no voters:
I20260929 02:46:53.476509 8529 raft_consensus.cc:2781] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:53.476637 8529 raft_consensus.cc:482] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:53.476732 8529 raft_consensus.cc:3037] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:53.479606 8529 raft_consensus.cc:504] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.480885 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acc5f1158b441a791daf16b111a17ca" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27"
I20260929 02:46:53.480942 8408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8acc5f1158b441a791daf16b111a17ca" candidate_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8489bd3d3e7b4ab496b480c999e453a9"
I20260929 02:46:53.481105 8274 raft_consensus.cc:3037] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:53.481138 8408 raft_consensus.cc:3037] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:53.481880 8529 leader_election.cc:288] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 election: Requested vote from peers 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571), 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549)
I20260929 02:46:53.483858 8408 raft_consensus.cc:2454] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 1.
I20260929 02:46:53.484210 8274 raft_consensus.cc:2454] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d4027cb24ba4cb48bfa9178ce3270d9 in term 1.
I20260929 02:46:53.484344 8062 leader_election.cc:302] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d4027cb24ba4cb48bfa9178ce3270d9, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:53.484663 8529 raft_consensus.cc:2781] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:53.484875 8529 raft_consensus.cc:686] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 LEADER]: Becoming Leader. State: Replica: 7d4027cb24ba4cb48bfa9178ce3270d9, State: Running, Role: LEADER
I20260929 02:46:53.485132 8529 consensus_queue.cc:237] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.487845 7971 catalog_manager.cc:5495] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } health_report { overall_health: UNKNOWN } } }
I20260929 02:46:53.524951 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.519907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.0.97.1:35129)
0929 02:46:53.520392 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.520396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.520411 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:46:53.520465 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.520469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.520528 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.520691 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:53.521056 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.521765 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.523134 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.523545 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.523677 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.524073 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:53.524188 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:53.524288 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:53.524431 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 02:46:53.524507 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":236,"threads_started":1}
I20260929 02:46:53.528689 8408 raft_consensus.cc:1261] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Refusing update from remote peer 7d4027cb24ba4cb48bfa9178ce3270d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:53.528746 8274 raft_consensus.cc:1261] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Refusing update from remote peer 7d4027cb24ba4cb48bfa9178ce3270d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:53.529451 8490 consensus_queue.cc:1035] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:53.529831 8529 consensus_queue.cc:1035] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:53.543891 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:53.544155 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:53.549578 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Sending DeleteTablet for 3 replicas of tablet 8acc5f1158b441a791daf16b111a17ca
I20260929 02:46:53.550231 8248 tablet_service.cc:1514] Processing DeleteTablet for tablet 8acc5f1158b441a791daf16b111a17ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52502
I20260929 02:46:53.550523 8555 tablet_replica.cc:330] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: stopping tablet replica
I20260929 02:46:53.550765 8555 raft_consensus.cc:2229] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:53.550984 8555 raft_consensus.cc:2258] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.551167 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet 8acc5f1158b441a791daf16b111a17ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53140
I20260929 02:46:53.551456 8556 tablet_replica.cc:330] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: stopping tablet replica
I20260929 02:46:53.551666 8556 raft_consensus.cc:2229] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:53.552217 8556 raft_consensus.cc:2258] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.553470 8556 ts_tablet_manager.cc:1905] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.555184 8248 tablet_service.cc:1514] Processing DeleteTablet for tablet 8acc5f1158b441a791daf16b111a17ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52502
I20260929 02:46:53.556237 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet 8acc5f1158b441a791daf16b111a17ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45656
I20260929 02:46:53.556627 8554 tablet_replica.cc:330] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: stopping tablet replica
I20260929 02:46:53.556857 8554 raft_consensus.cc:2229] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:53.557118 8554 raft_consensus.cc:2258] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.558162 7971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:46:53.558894 7971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:46:53.560858 8554 ts_tablet_manager.cc:1905] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.564980 8556 ts_tablet_manager.cc:1918] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:53.565207 8556 log.cc:1196] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/8acc5f1158b441a791daf16b111a17ca
I20260929 02:46:53.565349 8555 ts_tablet_manager.cc:1905] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.565800 8556 ts_tablet_manager.cc:1939] T 8acc5f1158b441a791daf16b111a17ca P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting consensus metadata
I20260929 02:46:53.569993 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce75c4d1cdf45819b6942c29789a014 (DEFAULT_TABLE table=test-table [id=8b229c5a618b409e80dd2d2b0e423ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:53.570463 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce75c4d1cdf45819b6942c29789a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:53.571642 8555 ts_tablet_manager.cc:1918] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:53.571799 8555 log.cc:1196] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/8acc5f1158b441a791daf16b111a17ca
I20260929 02:46:53.572211 8555 ts_tablet_manager.cc:1939] T 8acc5f1158b441a791daf16b111a17ca P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting consensus metadata
I20260929 02:46:53.573338 7960 catalog_manager.cc:4841] TS 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571): tablet 8acc5f1158b441a791daf16b111a17ca (table test-table [id=0edf82eb67154475b16b70d9f7712cc3]) successfully deleted
I20260929 02:46:53.573374 8248 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce75c4d1cdf45819b6942c29789a014 (DEFAULT_TABLE table=test-table [id=8b229c5a618b409e80dd2d2b0e423ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:53.573817 8248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce75c4d1cdf45819b6942c29789a014. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:46:53.574288 7960 catalog_manager.cc:4808] TS 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571): delete failed for tablet 8acc5f1158b441a791daf16b111a17ca because the tablet was not found. No further retry: Not found: Tablet not found: 8acc5f1158b441a791daf16b111a17ca
I20260929 02:46:53.576853 7957 catalog_manager.cc:4841] TS 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129): tablet 8acc5f1158b441a791daf16b111a17ca (table test-table [id=0edf82eb67154475b16b70d9f7712cc3]) successfully deleted
I20260929 02:46:53.577384 8110 tablet_service.cc:1467] Processing CreateTablet for tablet 0ce75c4d1cdf45819b6942c29789a014 (DEFAULT_TABLE table=test-table [id=8b229c5a618b409e80dd2d2b0e423ba5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:46:53.577845 8110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ce75c4d1cdf45819b6942c29789a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:53.585651 8554 ts_tablet_manager.cc:1918] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:46:53.585816 8554 log.cc:1196] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/8acc5f1158b441a791daf16b111a17ca
I20260929 02:46:53.586308 8554 ts_tablet_manager.cc:1939] T 8acc5f1158b441a791daf16b111a17ca P 8489bd3d3e7b4ab496b480c999e453a9: Deleting consensus metadata
I20260929 02:46:53.587488 7959 catalog_manager.cc:4841] TS 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549): tablet 8acc5f1158b441a791daf16b111a17ca (table test-table [id=0edf82eb67154475b16b70d9f7712cc3]) successfully deleted
I20260929 02:46:53.588670 8570 tablet_bootstrap.cc:492] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: Bootstrap starting.
I20260929 02:46:53.590371 8570 tablet_bootstrap.cc:654] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:53.592276 8570 tablet_bootstrap.cc:492] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: No bootstrap required, opened a new log
I20260929 02:46:53.592519 8570 ts_tablet_manager.cc:1397] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:46:53.593357 8570 raft_consensus.cc:348] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.593752 8570 raft_consensus.cc:374] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:53.593950 8571 tablet_bootstrap.cc:492] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: Bootstrap starting.
I20260929 02:46:53.594077 8570 raft_consensus.cc:729] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8489bd3d3e7b4ab496b480c999e453a9, State: Initialized, Role: FOLLOWER
I20260929 02:46:53.594435 8570 consensus_queue.cc:260] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.595731 8571 tablet_bootstrap.cc:654] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:53.595773 8570 ts_tablet_manager.cc:1428] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:53.599609 8572 tablet_bootstrap.cc:492] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: Bootstrap starting.
I20260929 02:46:53.600840 8571 tablet_bootstrap.cc:492] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: No bootstrap required, opened a new log
I20260929 02:46:53.601009 8571 ts_tablet_manager.cc:1397] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260929 02:46:53.601244 8572 tablet_bootstrap.cc:654] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:53.601770 8571 raft_consensus.cc:348] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.602051 8571 raft_consensus.cc:374] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:53.602137 8571 raft_consensus.cc:729] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4027cb24ba4cb48bfa9178ce3270d9, State: Initialized, Role: FOLLOWER
I20260929 02:46:53.602836 8571 consensus_queue.cc:260] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.603338 8572 tablet_bootstrap.cc:492] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: No bootstrap required, opened a new log
I20260929 02:46:53.603557 8572 ts_tablet_manager.cc:1397] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260929 02:46:53.604380 8572 raft_consensus.cc:348] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.604779 8572 raft_consensus.cc:374] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:53.604905 8572 raft_consensus.cc:729] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4254ef0b0d4e4ea6a6f01d4011786c27, State: Initialized, Role: FOLLOWER
I20260929 02:46:53.605226 8572 consensus_queue.cc:260] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.605861 8572 ts_tablet_manager.cc:1428] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:46:53.608803 8571 ts_tablet_manager.cc:1428] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260929 02:46:53.610941 8514 raft_consensus.cc:482] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:46:53.611126 8514 raft_consensus.cc:504] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.611851 8514 leader_election.cc:288] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571), 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129)
I20260929 02:46:53.612605 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce75c4d1cdf45819b6942c29789a014" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" is_pre_election: true
I20260929 02:46:53.612614 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce75c4d1cdf45819b6942c29789a014" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" is_pre_election: true
I20260929 02:46:53.612905 8134 raft_consensus.cc:2454] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 0.
I20260929 02:46:53.612907 8274 raft_consensus.cc:2454] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 0.
I20260929 02:46:53.613353 8342 leader_election.cc:302] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4254ef0b0d4e4ea6a6f01d4011786c27, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:53.613631 8514 raft_consensus.cc:2781] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:46:53.613762 8514 raft_consensus.cc:482] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:46:53.613862 8514 raft_consensus.cc:3037] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:53.617036 8514 raft_consensus.cc:504] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.618125 8274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce75c4d1cdf45819b6942c29789a014" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27"
I20260929 02:46:53.618353 8274 raft_consensus.cc:3037] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:53.621433 8274 raft_consensus.cc:2454] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 1.
I20260929 02:46:53.621883 8342 leader_election.cc:302] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4254ef0b0d4e4ea6a6f01d4011786c27, 8489bd3d3e7b4ab496b480c999e453a9; no voters:
I20260929 02:46:53.622608 8584 raft_consensus.cc:2781] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:53.623019 8514 leader_election.cc:288] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [CANDIDATE]: Term 1 election: Requested vote from peers 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571), 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129)
I20260929 02:46:53.623136 8134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ce75c4d1cdf45819b6942c29789a014" candidate_uuid: "8489bd3d3e7b4ab496b480c999e453a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9"
I20260929 02:46:53.623381 8134 raft_consensus.cc:3037] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:53.626550 8134 raft_consensus.cc:2454] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8489bd3d3e7b4ab496b480c999e453a9 in term 1.
I20260929 02:46:53.630125 8584 raft_consensus.cc:686] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 LEADER]: Becoming Leader. State: Replica: 8489bd3d3e7b4ab496b480c999e453a9, State: Running, Role: LEADER
I20260929 02:46:53.630609 8584 consensus_queue.cc:237] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } }
I20260929 02:46:53.633616 7971 catalog_manager.cc:5495] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8489bd3d3e7b4ab496b480c999e453a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8489bd3d3e7b4ab496b480c999e453a9" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44549 } health_report { overall_health: HEALTHY } } }
I20260929 02:46:53.664300 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.656756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45676 (local address 127.0.97.3:44549)
0929 02:46:53.657846 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.657853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.657877 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:46:53.657948 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.657954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.658049 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.658158 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:46:53.658624 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.659943 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.661940 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.662658 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.662797 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.663350 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:46:53.663526 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:46:53.663637 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:53.663790 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 02:46:53.663881 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":943,"thread_start_us":800,"threads_started":1}
I20260929 02:46:53.676638 8134 raft_consensus.cc:1261] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Refusing update from remote peer 8489bd3d3e7b4ab496b480c999e453a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:53.676644 8274 raft_consensus.cc:1261] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Refusing update from remote peer 8489bd3d3e7b4ab496b480c999e453a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:46:53.677338 8584 consensus_queue.cc:1035] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4254ef0b0d4e4ea6a6f01d4011786c27" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:53.677809 8585 consensus_queue.cc:1035] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4027cb24ba4cb48bfa9178ce3270d9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:46:53.693332 8526 mvcc.cc:205] Tried to move back new op lower bound from 7334502456013234176 to 7334502455834062848. Current Snapshot: MvccSnapshot[applied={T|T < 7334502456013234176}]
I20260929 02:46:53.762729 7971 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:53.762931 7971 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:46:53.765821 7971 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d41e939183ad4d4b808384c85c9e8fe3: Sending DeleteTablet for 3 replicas of tablet 0ce75c4d1cdf45819b6942c29789a014
I20260929 02:46:53.766747 8110 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce75c4d1cdf45819b6942c29789a014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53140
I20260929 02:46:53.766958 8248 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce75c4d1cdf45819b6942c29789a014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52502
I20260929 02:46:53.767081 8556 tablet_replica.cc:330] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: stopping tablet replica
I20260929 02:46:53.767258 8556 raft_consensus.cc:2229] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:53.767258 8389 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ce75c4d1cdf45819b6942c29789a014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:46:53 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45656
I20260929 02:46:53.767526 8556 raft_consensus.cc:2258] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.767557 8555 tablet_replica.cc:330] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: stopping tablet replica
I20260929 02:46:53.767721 8555 raft_consensus.cc:2229] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:46:53.767970 8555 raft_consensus.cc:2258] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.768158 8554 tablet_replica.cc:330] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: stopping tablet replica
I20260929 02:46:53.768338 8554 raft_consensus.cc:2229] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:46:53.768405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 8044
I20260929 02:46:53.768738 8554 raft_consensus.cc:2258] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:46:53.770231 8556 ts_tablet_manager.cc:1905] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.770263 8554 ts_tablet_manager.cc:1905] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.773586 8555 ts_tablet_manager.cc:1905] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:46:53.777513 8556 ts_tablet_manager.cc:1918] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:53.777668 8556 log.cc:1196] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/0ce75c4d1cdf45819b6942c29789a014
I20260929 02:46:53.778120 8556 ts_tablet_manager.cc:1939] T 0ce75c4d1cdf45819b6942c29789a014 P 7d4027cb24ba4cb48bfa9178ce3270d9: Deleting consensus metadata
I20260929 02:46:53.779263 7957 catalog_manager.cc:4841] TS 7d4027cb24ba4cb48bfa9178ce3270d9 (127.0.97.1:35129): tablet 0ce75c4d1cdf45819b6942c29789a014 (table test-table [id=8b229c5a618b409e80dd2d2b0e423ba5]) successfully deleted
I20260929 02:46:53.780865 8554 ts_tablet_manager.cc:1918] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:53.781000 8554 log.cc:1196] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/0ce75c4d1cdf45819b6942c29789a014
I20260929 02:46:53.781453 8554 ts_tablet_manager.cc:1939] T 0ce75c4d1cdf45819b6942c29789a014 P 8489bd3d3e7b4ab496b480c999e453a9: Deleting consensus metadata
I20260929 02:46:53.782521 7959 catalog_manager.cc:4841] TS 8489bd3d3e7b4ab496b480c999e453a9 (127.0.97.3:44549): tablet 0ce75c4d1cdf45819b6942c29789a014 (table test-table [id=8b229c5a618b409e80dd2d2b0e423ba5]) successfully deleted
I20260929 02:46:53.783247 8555 ts_tablet_manager.cc:1918] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 02:46:53.783419 8555 log.cc:1196] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/0ce75c4d1cdf45819b6942c29789a014
I20260929 02:46:53.783779 8555 ts_tablet_manager.cc:1939] T 0ce75c4d1cdf45819b6942c29789a014 P 4254ef0b0d4e4ea6a6f01d4011786c27: Deleting consensus metadata
I20260929 02:46:53.784740 7960 catalog_manager.cc:4841] TS 4254ef0b0d4e4ea6a6f01d4011786c27 (127.0.97.2:40571): tablet 0ce75c4d1cdf45819b6942c29789a014 (table test-table [id=8b229c5a618b409e80dd2d2b0e423ba5]) successfully deleted
Sep 29 02:46:53 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260929 02:46:53.801255 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.769670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.0.97.1:35129)
0929 02:46:53.769829 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.769835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.785133 (+ 15298us) server_negotiation.cc:407] Connection header received
0929 02:46:53.785265 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.785270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.785358 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.785480 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:53.786701 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.787929 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.789301 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.789971 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.790105 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.794131 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:53.794151 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:53.794172 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:53.794222 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:53.798278 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:53.799015 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:53.799026 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:53.799030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:53.799190 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:53.799571 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:53.799577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:53.799580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:53.800026 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:53.800349 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:53.800615 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 02:46:53.800856 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 02:46:53.802067 8175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:53.948164 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 8044
I20260929 02:46:53.971623 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 8184
Sep 29 02:46:53 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260929 02:46:53.987211 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:53.972310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52524 (local address 127.0.97.2:40571)
0929 02:46:53.972949 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:53.972953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:53.972971 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:46:53.973024 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:53.973028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:53.973090 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:53.973210 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:53.974422 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.975223 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.976428 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:53.977102 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:53.977226 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:53.981304 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:53.981318 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:53.981334 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:53.981366 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:53.984221 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:53.985008 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:53.985016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:53.985018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:53.985144 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:53.985572 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:53.985576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:53.985578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:53.985969 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:53.986197 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:53.986598 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 02:46:53.986869 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":338,"threads_started":1}
I20260929 02:46:53.987929 8314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:54.130328 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 8184
I20260929 02:46:54.151836 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 8323
Sep 29 02:46:54 dist-test-slave-6tjh krb5kdc[7928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260929 02:46:54.166710 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:54.152581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.0.97.3:44549)
0929 02:46:54.152809 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:54.152813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:54.152832 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:46:54.152907 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:54.152911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:54.152981 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:54.153059 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:54.154399 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:54.155137 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:54.156344 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:54.156840 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:54.156948 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:54.161321 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:54.161336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:54.161352 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:54.161402 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:54.164227 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:54.164909 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:54.164916 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:54.164919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:54.165040 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:54.165351 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:54.165354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:54.165356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:54.165692 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:54.165920 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:54.166168 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 02:46:54.166337 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:46:54.167308 8454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:54.291858 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 8323
I20260929 02:46:54.313372 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 7941
I20260929 02:46:54.314611 8013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:46:54.428787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 7941
2026-09-29T02:46:54Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5982 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:54 dist-test-slave-6tjh krb5kdc[8601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:54 dist-test-slave-6tjh krb5kdc[8601](info): set up 2 sockets
Sep 29 02:46:54 dist-test-slave-6tjh krb5kdc[8601](info): commencing operation
krb5kdc: starting...
W20260929 02:46:55.646270 388 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:55 dist-test-slave-6tjh krb5kdc[8601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:55.884992 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33835
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:56.213721 8614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:56.214082 8614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:56.214257 8614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:56.221808 8614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:56.221966 8614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:56.222076 8614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:56.222175 8614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:56.222267 8614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:56.232322 8614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:56.233692 8614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:56.235018 8614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:56.246935 8619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:56.247341 8622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:56.246917 8620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:56.247694 8614 server_base.cc:1034] running on GCE node
I20260929 02:46:56.248513 8614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:56.249998 8614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:56.251274 8614 hybrid_clock.cc:648] HybridClock initialized: now 1790650016251199 us; error 53 us; skew 500 ppm
I20260929 02:46:56.251669 8614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T02:46:56Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1894 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:46:56 dist-test-slave-6tjh krb5kdc[8628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:46:56 dist-test-slave-6tjh krb5kdc[8628](info): set up 2 sockets
Sep 29 02:46:56 dist-test-slave-6tjh krb5kdc[8628](info): commencing operation
krb5kdc: starting...
W20260929 02:46:57.469545 388 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.006s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:46:57 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:46:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:46:57.697398 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39313
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:58.027557 8641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:58.027887 8641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:58.028051 8641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:58.035436 8641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:46:58.035560 8641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:58.035645 8641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:58.035742 8641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:46:58.035835 8641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:46:58.045766 8641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:58.047160 8641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:58.048488 8641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:58.061131 8649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:58.061100 8647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:58.061348 8646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:58.061918 8641 server_base.cc:1034] running on GCE node
I20260929 02:46:58.062741 8641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:58.064229 8641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:58.065485 8641 hybrid_clock.cc:648] HybridClock initialized: now 1790650018065428 us; error 73 us; skew 500 ppm
I20260929 02:46:58.065933 8641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:58 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:58.069514 8641 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:46:58.071190 8641 webserver.cc:460] Webserver started at http://127.0.97.62:45249/ using document root <none> and password file <none>
I20260929 02:46:58.071902 8641 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:58.072046 8641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:58.072365 8641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:58.075469 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "ff6558977c734fe09f042a550b226f63"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "e4b44481c9c7494a4f881aca0771543e"
server_key_iv: "6e2010e7bb6b3ad66b84b18ca65589b1"
server_key_version: "encryptionkey@0"
I20260929 02:46:58.076282 8641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "ff6558977c734fe09f042a550b226f63"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "e4b44481c9c7494a4f881aca0771543e"
server_key_iv: "6e2010e7bb6b3ad66b84b18ca65589b1"
server_key_version: "encryptionkey@0"
I20260929 02:46:58.081697 8641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:46:58.085660 8656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:58.087179 8641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:46:58.087422 8641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "ff6558977c734fe09f042a550b226f63"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "e4b44481c9c7494a4f881aca0771543e"
server_key_iv: "6e2010e7bb6b3ad66b84b18ca65589b1"
server_key_version: "encryptionkey@0"
I20260929 02:46:58.087664 8641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:58.109299 8641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:58.114033 8641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:58.114384 8641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:58.136102 8641 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39313
I20260929 02:46:58.136132 8707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39313 every 8 connection(s)
I20260929 02:46:58.137631 8641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:46:58.138221 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 8641
I20260929 02:46:58.138445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:46:58.139295 388 external_mini_cluster.cc:1428] Setting key ce9e6eabe3ed636065a230e02d5b7e14
I20260929 02:46:58.142238 8708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:46:58.152823 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Bootstrap starting.
Sep 29 02:46:58 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:58.156287 8708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Neither blocks nor log segments found. Creating new log.
I20260929 02:46:58.157406 8708 log.cc:824] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Log is configured to *not* fsync() on all Append() calls
I20260929 02:46:58.160636 8708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: No bootstrap required, opened a new log
I20260929 02:46:58.161861 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:58.141359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51404 (local address 127.0.97.62:39313)
0929 02:46:58.142166 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:58.142179 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:58.142223 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:46:58.143988 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:58.144012 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:58.144394 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:58.144901 (+ 507us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:58.146374 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:58.147752 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:58.149123 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:58.149844 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:58.150076 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:58.154277 (+ 4201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:58.154284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:58.154299 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:58.154334 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:58.157624 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:58.158421 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:58.158429 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:58.158437 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:58.158648 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:58.159116 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:58.159120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:58.159122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:58.159611 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:58.159851 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:58.160289 (+ 438us) server_negotiation.cc:299] Negotiation successful
0929 02:46:58.160674 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":400,"threads_started":1}
I20260929 02:46:58.168826 8708 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:46:58.169207 8708 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:46:58.169322 8708 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6558977c734fe09f042a550b226f63, State: Initialized, Role: FOLLOWER
I20260929 02:46:58.169961 8708 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:46:58.170248 8708 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:46:58.170398 8708 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:46:58.170580 8708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:46:58.172828 8708 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:46:58.173400 8708 leader_election.cc:302] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff6558977c734fe09f042a550b226f63; no voters:
I20260929 02:46:58.174023 8708 leader_election.cc:288] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:46:58.174172 8713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:46:58.174911 8713 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 LEADER]: Becoming Leader. State: Replica: ff6558977c734fe09f042a550b226f63, State: Running, Role: LEADER
I20260929 02:46:58.175541 8713 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:46:58.176743 8708 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:46:58.179096 8715 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6558977c734fe09f042a550b226f63. Latest consensus state: current_term: 1 leader_uuid: "ff6558977c734fe09f042a550b226f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } } }
I20260929 02:46:58.179081 8714 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff6558977c734fe09f042a550b226f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } } }
I20260929 02:46:58.179668 8715 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:58.179788 8714 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: This master's current role is: LEADER
I20260929 02:46:58.183141 8718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:46:58.188133 8718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:46:58.197438 8718 catalog_manager.cc:1347] Generated new cluster ID: d54e3ce643244bc1bbaee9a9c8f80ee7
I20260929 02:46:58.197580 8718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:46:58.215070 8718 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:46:58.216455 8718 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:46:58.225612 8718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Generated new TSK 0
I20260929 02:46:58.226774 8718 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:46:58.289091 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39313
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:58.628376 8734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:58.628782 8734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:58.628970 8734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:58.636230 8734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:58.636405 8734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:58.636662 8734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:46:58.646518 8734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:58.647903 8734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:58.649212 8734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:58.662331 8739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:58.662621 8740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:58.662644 8742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:58.663864 8734 server_base.cc:1034] running on GCE node
I20260929 02:46:58.664420 8734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:58.665446 8734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:58.666663 8734 hybrid_clock.cc:648] HybridClock initialized: now 1790650018666611 us; error 46 us; skew 500 ppm
I20260929 02:46:58.666968 8734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:58 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:58.674780 8734 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:46:58.676326 8734 webserver.cc:460] Webserver started at http://127.0.97.1:43341/ using document root <none> and password file <none>
I20260929 02:46:58.677163 8734 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:58.677338 8734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:58.677675 8734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:58.680661 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "28c1bc335bcc4599882f2989af3d4757"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "d35dad4da78526ba8e870fdacdeaf35f"
server_key_iv: "7298e3e6c97c98ea1acfa734637eff91"
server_key_version: "encryptionkey@0"
I20260929 02:46:58.681447 8734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "28c1bc335bcc4599882f2989af3d4757"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "d35dad4da78526ba8e870fdacdeaf35f"
server_key_iv: "7298e3e6c97c98ea1acfa734637eff91"
server_key_version: "encryptionkey@0"
I20260929 02:46:58.686901 8734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:46:58.690582 8749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:58.692003 8734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:46:58.692168 8734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "28c1bc335bcc4599882f2989af3d4757"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "d35dad4da78526ba8e870fdacdeaf35f"
server_key_iv: "7298e3e6c97c98ea1acfa734637eff91"
server_key_version: "encryptionkey@0"
I20260929 02:46:58.692395 8734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:58.726996 8734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:58.731751 8734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:58.732003 8734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:58.733009 8734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:58.734328 8734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:58.734400 8734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:58.734488 8734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:58.734535 8734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:58.767539 8734 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41427
I20260929 02:46:58.767635 8861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41427 every 8 connection(s)
I20260929 02:46:58.768831 8734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:46:58.772076 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 8734
I20260929 02:46:58.772291 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:46:58.773204 388 external_mini_cluster.cc:1428] Setting key f97787678daf0c90a4ad25f0e7c0d975
Sep 29 02:46:58 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:58.792972 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:58.771585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55077 (local address 127.0.97.62:39313)
0929 02:46:58.772106 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:58.772113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:58.773384 (+ 1271us) server_negotiation.cc:407] Connection header received
0929 02:46:58.775527 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:58.775531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:58.775592 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:58.775710 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:58.777894 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:58.778687 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:58.781252 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:58.781657 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:58.781766 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:58.786071 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:58.786080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:58.786089 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:58.786136 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:58.789521 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:58.790320 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:58.790328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:58.790335 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:58.790484 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:58.790907 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:58.790915 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:58.790921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:58.791291 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:58.791444 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:58.792168 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 02:46:58.792425 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":271,"threads_started":1}
I20260929 02:46:58.793774 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:58.772160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:55077)
0929 02:46:58.773118 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:58.773171 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:58.775197 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:58.775987 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:58.775999 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:58.776826 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:58.777710 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:58.777737 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:58.778857 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:58.778865 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:58.781071 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:58.781090 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:58.781895 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:58.781904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:58.782276 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:58.783188 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:58.783220 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:58.785816 (+ 2596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:58.789684 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:58.789695 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:58.789714 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:58.790177 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:58.790589 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:58.790593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:58.790595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:58.790784 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:58.791407 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:58.791413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:58.791742 (+ 329us) client_negotiation.cc:769] Sending connection context
0929 02:46:58.791994 (+ 252us) client_negotiation.cc:240] Negotiation successful
0929 02:46:58.792522 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":381,"threads_started":1}
I20260929 02:46:58.795325 8862 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39313
I20260929 02:46:58.795660 8862 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:58.796600 8862 heartbeater.cc:511] Master 127.0.97.62:39313 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:46:58.798868 8673 ts_manager.cc:194] Registered new tserver with Master: 28c1bc335bcc4599882f2989af3d4757 (127.0.97.1:41427)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:46:58.800570 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:55077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:46:58.828104 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39313
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:59.151875 8868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:59.152133 8868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:59.152295 8868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:59.159545 8868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:59.159642 8868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:59.159830 8868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:46:59.169384 8868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:59.170682 8868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:59.172012 8868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:59.185156 8874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:59.185125 8876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:59.185715 8873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:59.186197 8868 server_base.cc:1034] running on GCE node
I20260929 02:46:59.186848 8868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:59.187811 8868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:59.189031 8868 hybrid_clock.cc:648] HybridClock initialized: now 1790650019188982 us; error 63 us; skew 500 ppm
I20260929 02:46:59.189432 8868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:59 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:59.193087 8868 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:46:59.194631 8868 webserver.cc:460] Webserver started at http://127.0.97.2:36105/ using document root <none> and password file <none>
I20260929 02:46:59.195329 8868 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:59.195462 8868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:59.195812 8868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:59.198829 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "d2822808149c4bb893f8dbcdc5be6ac8"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "67df35bf4f82a6a7d297cf0f960b7276"
server_key_iv: "e5a9a6380120a2ea3bf2894e60922493"
server_key_version: "encryptionkey@0"
I20260929 02:46:59.199618 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d2822808149c4bb893f8dbcdc5be6ac8"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "67df35bf4f82a6a7d297cf0f960b7276"
server_key_iv: "e5a9a6380120a2ea3bf2894e60922493"
server_key_version: "encryptionkey@0"
I20260929 02:46:59.205108 8868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:46:59.209029 8883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:59.210650 8868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:46:59.210891 8868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "d2822808149c4bb893f8dbcdc5be6ac8"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "67df35bf4f82a6a7d297cf0f960b7276"
server_key_iv: "e5a9a6380120a2ea3bf2894e60922493"
server_key_version: "encryptionkey@0"
I20260929 02:46:59.211134 8868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:59.229349 8868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:59.233860 8868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:59.234169 8868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:59.235188 8868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:59.236644 8868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:59.236763 8868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:59.236932 8868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:59.237012 8868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:59.271502 8868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36197
I20260929 02:46:59.271546 8995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36197 every 8 connection(s)
I20260929 02:46:59.273013 8868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:46:59.279259 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 8868
I20260929 02:46:59.279439 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:46:59.279978 388 external_mini_cluster.cc:1428] Setting key 4df51f9565a88c8df8bde525bc21585c
Sep 29 02:46:59 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:59.293406 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:59.275568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51457 (local address 127.0.97.62:39313)
0929 02:46:59.275737 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:59.275741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:59.277028 (+ 1287us) server_negotiation.cc:407] Connection header received
0929 02:46:59.278955 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:59.278959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:59.279032 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:59.279127 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:59.281122 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.281849 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.283557 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.283989 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.284126 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:59.287582 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:59.287587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:59.287590 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:59.287621 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:59.290268 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:59.291012 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:59.291017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:59.291019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:59.291144 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:59.291584 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:59.291595 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:59.291597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:59.291928 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:59.292079 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:59.292837 (+ 758us) server_negotiation.cc:299] Negotiation successful
0929 02:46:59.293063 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:46:59.294440 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:59.276063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:51457)
0929 02:46:59.276850 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:59.276892 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:59.278663 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:59.279441 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:59.279455 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:59.280182 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:59.280959 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:59.280977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.282008 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.282014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:59.283422 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:59.283433 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.284213 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.284220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:59.284456 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:59.285208 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:59.285231 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:59.287343 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:59.290422 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:59.290431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:59.290450 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:59.290872 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:59.291258 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:59.291263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:59.291267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:59.291466 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:59.292054 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:59.292063 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:59.292414 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 02:46:59.292739 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 02:46:59.293311 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":312,"threads_started":1}
I20260929 02:46:59.296023 8996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39313
I20260929 02:46:59.296340 8996 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:59.297202 8996 heartbeater.cc:511] Master 127.0.97.62:39313 requested a full tablet report, sending...
I20260929 02:46:59.298651 8673 ts_manager.cc:194] Registered new tserver with Master: d2822808149c4bb893f8dbcdc5be6ac8 (127.0.97.2:36197)
I20260929 02:46:59.299845 8673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51457
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:46:59.328456 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39313
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:46:59.650416 9001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:46:59.650746 9001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:46:59.650931 9001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:46:59.658212 9001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:46:59.658358 9001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:46:59.658545 9001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:46:59.668429 9001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:46:59.669808 9001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:46:59.671133 9001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:46:59.684214 9007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:59.684149 9009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:46:59.684190 9006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:46:59.684932 9001 server_base.cc:1034] running on GCE node
I20260929 02:46:59.685504 9001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:46:59.686331 9001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:46:59.687577 9001 hybrid_clock.cc:648] HybridClock initialized: now 1790650019687511 us; error 53 us; skew 500 ppm
I20260929 02:46:59.687942 9001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:46:59 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:46:59.691026 9001 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:46:59.692477 9001 webserver.cc:460] Webserver started at http://127.0.97.3:34615/ using document root <none> and password file <none>
I20260929 02:46:59.693169 9001 fs_manager.cc:359] Metadata directory not provided
I20260929 02:46:59.693298 9001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:46:59.693601 9001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:46:59.696532 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "e99fc0e29ee54b0ba4423fa1b93a7e92"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "561640b51ee22bdc037964f2fbeea473"
server_key_iv: "8c7bd92d7975cd2f4fa6ae8233c8d204"
server_key_version: "encryptionkey@0"
I20260929 02:46:59.697638 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e99fc0e29ee54b0ba4423fa1b93a7e92"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "561640b51ee22bdc037964f2fbeea473"
server_key_iv: "8c7bd92d7975cd2f4fa6ae8233c8d204"
server_key_version: "encryptionkey@0"
I20260929 02:46:59.702700 9001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:46:59.706187 9016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:59.707538 9001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 02:46:59.707705 9001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "e99fc0e29ee54b0ba4423fa1b93a7e92"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "561640b51ee22bdc037964f2fbeea473"
server_key_iv: "8c7bd92d7975cd2f4fa6ae8233c8d204"
server_key_version: "encryptionkey@0"
I20260929 02:46:59.707901 9001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:46:59.746428 9001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:46:59.750976 9001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:46:59.751286 9001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:46:59.752269 9001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:46:59.753600 9001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:46:59.753679 9001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:59.753861 9001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:46:59.753912 9001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:46:59.789842 9001 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39845
I20260929 02:46:59.789949 9128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39845 every 8 connection(s)
I20260929 02:46:59.791146 9001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:46:59.799930 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9001
I20260929 02:46:59.800063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:46:59.800607 388 external_mini_cluster.cc:1428] Setting key 7c3c6a9f34c801f629534ed8d1c48e59
I20260929 02:46:59.803628 8862 heartbeater.cc:503] Master 127.0.97.62:39313 was elected leader, sending a full tablet report...
Sep 29 02:46:59 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650019, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:46:59.813103 9131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:59.794005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49345 (local address 127.0.97.62:39313)
0929 02:46:59.794434 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:59.794439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:59.795829 (+ 1390us) server_negotiation.cc:407] Connection header received
0929 02:46:59.797749 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:59.797754 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:59.797827 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:59.797940 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:59.799415 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.800342 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.802540 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.803135 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.803268 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:59.807393 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:59.807398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:59.807401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:59.807430 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:59.810050 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:59.810865 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:59.810869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:59.810871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:59.810983 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:59.811403 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:59.811406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:59.811408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:59.811646 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:59.811750 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:59.812672 (+ 922us) server_negotiation.cc:299] Negotiation successful
0929 02:46:59.812839 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260929 02:46:59.814174 9132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:59.794547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:49345)
0929 02:46:59.795634 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0929 02:46:59.795680 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:46:59.797482 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:46:59.798150 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:46:59.798158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:46:59.798671 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:46:59.799236 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:59.799257 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.800488 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.800494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:59.802370 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:46:59.802386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.803427 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.803436 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:46:59.803758 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:59.804593 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:46:59.804639 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:46:59.807186 (+ 2547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:46:59.810182 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:59.810191 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:59.810209 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:59.810698 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:59.811092 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:46:59.811096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:46:59.811098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:46:59.811301 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:46:59.811861 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:46:59.811870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:46:59.812274 (+ 404us) client_negotiation.cc:769] Sending connection context
0929 02:46:59.812534 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 02:46:59.813035 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":797,"thread_start_us":586,"threads_started":1}
I20260929 02:46:59.815649 9129 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39313
I20260929 02:46:59.815999 9129 heartbeater.cc:464] Registering TS with master...
I20260929 02:46:59.816937 9129 heartbeater.cc:511] Master 127.0.97.62:39313 requested a full tablet report, sending...
I20260929 02:46:59.818317 8672 ts_manager.cc:194] Registered new tserver with Master: e99fc0e29ee54b0ba4423fa1b93a7e92 (127.0.97.3:39845)
I20260929 02:46:59.819252 8672 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:49345
I20260929 02:46:59.829300 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260929 02:46:59.969555 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 8734
Sep 29 02:46:59 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:46:59.986485 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:46:59.970825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43188 (local address 127.0.97.1:41427)
0929 02:46:59.971534 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0929 02:46:59.971541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:46:59.971568 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:46:59.971627 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:46:59.971633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:46:59.971864 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:46:59.971995 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:46:59.973513 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.974509 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.975836 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:46:59.976379 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:46:59.976472 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:46:59.980980 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:46:59.980990 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:46:59.981024 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:46:59.981060 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:46:59.983957 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:59.984637 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:59.984644 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:59.984648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:59.984783 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:46:59.985104 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:46:59.985107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:46:59.985109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:46:59.985407 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:46:59.985628 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 02:46:59.985925 (+ 297us) server_negotiation.cc:299] Negotiation successful
0929 02:46:59.986101 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":410,"threads_started":1}
I20260929 02:46:59.987715 8857 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:00.069839 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 8734
I20260929 02:47:00.087431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 8868
Sep 29 02:47:00 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:47:00.104003 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:00.088371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55454 (local address 127.0.97.2:36197)
0929 02:47:00.088984 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:00.088991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:00.089027 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:47:00.089096 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:00.089103 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:00.089317 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:00.089465 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:00.090790 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:00.092085 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:00.093636 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:00.094264 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:00.094373 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:00.098665 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:00.098677 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:00.098691 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:00.098727 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:00.101611 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:00.102248 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:00.102254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:00.102256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:00.102365 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:00.102661 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:00.102664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:00.102666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:00.102994 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:00.103211 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:00.103513 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 02:47:00.103697 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":331,"threads_started":1}
I20260929 02:47:00.105356 8991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:00.189008 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 8868
I20260929 02:47:00.204641 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9001
Sep 29 02:47:00 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:47:00.221782 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:00.205790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.0.97.3:39845)
0929 02:47:00.206306 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:00.206313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:00.206345 (+ 32us) server_negotiation.cc:407] Connection header received
0929 02:47:00.206407 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:00.206413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:00.206620 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:00.206753 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:00.208050 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:00.209138 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:00.210444 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:00.211111 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:00.211219 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:00.215830 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:00.215842 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:00.215858 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:00.215901 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:00.218832 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:00.219504 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:00.219511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:00.219513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:00.219632 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:00.220016 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:00.220019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:00.220021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:00.220368 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:00.220664 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:00.221186 (+ 522us) server_negotiation.cc:299] Negotiation successful
0929 02:47:00.221399 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260929 02:47:00.223174 9124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:00.305711 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9001
I20260929 02:47:00.321501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 8641
I20260929 02:47:00.322638 8703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:00.423890 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 8641
I20260929 02:47:00.444270 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39313
--webserver_interface=127.0.97.62
--webserver_port=45249
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:47:00.769640 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:00.769964 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:00.770052 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:00.777217 9147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:47:00.777330 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:00.777376 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:00.777433 9147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:47:00.777475 9147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:47:00.787405 9147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=45249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:00.788796 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:00.790131 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:00.801674 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:00.801759 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:00.801739 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:00.802537 9147 server_base.cc:1034] running on GCE node
I20260929 02:47:00.803360 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:00.804795 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:00.806033 9147 hybrid_clock.cc:648] HybridClock initialized: now 1790650020805983 us; error 56 us; skew 500 ppm
I20260929 02:47:00.806404 9147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:00 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:00.809470 9147 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:47:00.810973 9147 webserver.cc:460] Webserver started at http://127.0.97.62:45249/ using document root <none> and password file <none>
I20260929 02:47:00.811638 9147 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:00.811801 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:00.817044 9147 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:47:00.819844 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:00.821170 9147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:47:00.821352 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "ff6558977c734fe09f042a550b226f63"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "e4b44481c9c7494a4f881aca0771543e"
server_key_iv: "6e2010e7bb6b3ad66b84b18ca65589b1"
server_key_version: "encryptionkey@0"
I20260929 02:47:00.822281 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:00.853236 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:00.857364 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:00.857661 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:00.876291 9147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39313
I20260929 02:47:00.876312 9213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39313 every 8 connection(s)
I20260929 02:47:00.877799 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:47:00.884761 9214 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:47:00.885290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9147
I20260929 02:47:00.886060 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41427
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:39313
--webserver_port=43341
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:47:00.888130 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Bootstrap starting.
I20260929 02:47:00.906719 9214 log.cc:824] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Log is configured to *not* fsync() on all Append() calls
I20260929 02:47:00.915148 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:47:00.915664 9214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Bootstrap complete.
I20260929 02:47:00.924463 9214 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:47:00.925460 9214 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6558977c734fe09f042a550b226f63, State: Initialized, Role: FOLLOWER
I20260929 02:47:00.926179 9214 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:47:00.926436 9214 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:47:00.926559 9214 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:47:00.926767 9214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:47:00.930672 9214 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:47:00.931187 9214 leader_election.cc:302] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff6558977c734fe09f042a550b226f63; no voters:
I20260929 02:47:00.931861 9214 leader_election.cc:288] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:47:00.931986 9218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:47:00.933420 9218 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [term 2 LEADER]: Becoming Leader. State: Replica: ff6558977c734fe09f042a550b226f63, State: Running, Role: LEADER
I20260929 02:47:00.934031 9218 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } }
I20260929 02:47:00.934221 9214 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:47:00.936730 9219 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ff6558977c734fe09f042a550b226f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } } }
I20260929 02:47:00.937186 9219 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:00.941843 9220 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff6558977c734fe09f042a550b226f63. Latest consensus state: current_term: 2 leader_uuid: "ff6558977c734fe09f042a550b226f63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff6558977c734fe09f042a550b226f63" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39313 } } }
I20260929 02:47:00.942205 9220 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63 [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:00.944995 9224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:47:00.950937 9224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:47:00.952338 9224 catalog_manager.cc:1259] Loaded cluster ID: d54e3ce643244bc1bbaee9a9c8f80ee7
I20260929 02:47:00.952512 9224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:47:00.956331 9224 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:47:00.958096 9224 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ff6558977c734fe09f042a550b226f63: Loaded TSK: 0
I20260929 02:47:00.959563 9224 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:47:01.231902 9216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:01.232206 9216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:01.232355 9216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:01.239938 9216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:01.240053 9216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:01.240231 9216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:47:01.250293 9216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=43341
--tserver_master_addrs=127.0.97.62:39313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:01.251632 9216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:01.252964 9216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:01.267778 9241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:01.267778 9240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:01.268821 9243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:01.270427 9216 server_base.cc:1034] running on GCE node
I20260929 02:47:01.271025 9216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:01.271956 9216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:01.273178 9216 hybrid_clock.cc:648] HybridClock initialized: now 1790650021273127 us; error 45 us; skew 500 ppm
I20260929 02:47:01.273499 9216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:01.276929 9216 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260929 02:47:01.278571 9216 webserver.cc:460] Webserver started at http://127.0.97.1:43341/ using document root <none> and password file <none>
I20260929 02:47:01.279289 9216 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:01.279441 9216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:01.285151 9216 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:47:01.288117 9250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:01.289706 9216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:47:01.289897 9216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "28c1bc335bcc4599882f2989af3d4757"
format_stamp: "Formatted at 2026-09-29 02:46:58 on dist-test-slave-6tjh"
server_key: "d35dad4da78526ba8e870fdacdeaf35f"
server_key_iv: "7298e3e6c97c98ea1acfa734637eff91"
server_key_version: "encryptionkey@0"
I20260929 02:47:01.290864 9216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:01.307732 9216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:01.312016 9216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:01.312314 9216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:01.313355 9216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:01.314683 9216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:01.314771 9216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:01.314934 9216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:01.315033 9216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:01.349511 9216 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41427
I20260929 02:47:01.349545 9362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41427 every 8 connection(s)
I20260929 02:47:01.350941 9216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:47:01.358480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9216
I20260929 02:47:01.359244 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36197
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:39313
--webserver_port=36105
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:01.371019 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:01.353912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:37931)
0929 02:47:01.354749 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:01.354799 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:01.356520 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:01.358165 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:01.358181 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:01.358980 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:01.359852 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.359883 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.361800 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.361808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.363986 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.364005 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.365317 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.365326 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.365622 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.366394 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:01.366417 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:01.370068 (+ 3651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":362,"threads_started":1}
W20260929 02:47:01.371809 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:01.353862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37931 (local address 127.0.97.62:39313)
0929 02:47:01.354565 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:01.354576 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:01.354952 (+ 376us) server_negotiation.cc:407] Connection header received
0929 02:47:01.357174 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:01.357189 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:01.357563 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:01.357883 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:01.360094 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.361606 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.364202 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.365064 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.365225 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.370561 (+ 5336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37931: BlockingRecv error: recv got EOF from 127.0.97.1:37931 (error 108)
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":289,"threads_started":1}
W20260929 02:47:01.371977 9363 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39313 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39313: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:01.380425 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:01.372650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:43259)
0929 02:47:01.372814 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:01.372855 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:01.372961 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:01.373253 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:01.373257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:01.373578 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:01.373705 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.373713 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.374727 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.374733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.375641 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.375657 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.376877 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.376881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.377009 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.377639 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:01.377655 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:01.380233 (+ 2578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 02:47:01.380864 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:01.372797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43259 (local address 127.0.97.62:39313)
0929 02:47:01.372970 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:01.372975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:01.372995 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:47:01.373058 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:01.373063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:01.373130 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:01.373230 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:01.373842 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.374588 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.375820 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.376711 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.376887 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.380530 (+ 3643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:43259: BlockingRecv error: recv got EOF from 127.0.97.1:43259 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:01.388819 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:01.381315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:49087)
0929 02:47:01.381474 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:01.381498 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:01.381592 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:01.381966 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:01.381970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:01.382256 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:01.382389 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.382396 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.383346 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.383352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.384241 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.384251 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.385048 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.385054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.385221 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.385873 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:01.385895 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:01.388490 (+ 2595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 02:47:01.388988 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:01.381527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49087 (local address 127.0.97.62:39313)
0929 02:47:01.381708 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:01.381713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:01.381729 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:47:01.381777 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:01.381780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:01.381843 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:01.381945 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:01.382518 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.383202 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.384398 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.384904 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.385039 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.388725 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49087: BlockingRecv error: recv got EOF from 127.0.97.1:49087 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 02:47:01.389060 9363 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:47:01.696832 9367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:01.697103 9367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:01.697276 9367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:01.704366 9367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:01.704478 9367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:01.704685 9367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:47:01.714558 9367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36105
--tserver_master_addrs=127.0.97.62:39313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:01.715943 9367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:01.717387 9367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:01.734593 9372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:01.734663 9373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:01.734925 9375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:01.735936 9367 server_base.cc:1034] running on GCE node
I20260929 02:47:01.736630 9367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:01.737583 9367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:01.738828 9367 hybrid_clock.cc:648] HybridClock initialized: now 1790650021738778 us; error 46 us; skew 500 ppm
I20260929 02:47:01.739167 9367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650021, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:01.742401 9367 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260929 02:47:01.744061 9367 webserver.cc:460] Webserver started at http://127.0.97.2:36105/ using document root <none> and password file <none>
I20260929 02:47:01.744884 9367 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:01.745038 9367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:01.750964 9367 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:47:01.753831 9382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:01.755229 9367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:47:01.755445 9367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "d2822808149c4bb893f8dbcdc5be6ac8"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "67df35bf4f82a6a7d297cf0f960b7276"
server_key_iv: "e5a9a6380120a2ea3bf2894e60922493"
server_key_version: "encryptionkey@0"
I20260929 02:47:01.756412 9367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:01.785322 9367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:01.789793 9367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:01.790079 9367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:01.791103 9367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:01.792435 9367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:01.792521 9367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:01.792701 9367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:01.792768 9367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:01.826287 9367 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36197
I20260929 02:47:01.826381 9494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36197 every 8 connection(s)
I20260929 02:47:01.827777 9367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:47:01.833952 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9367
I20260929 02:47:01.834633 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:39845
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:39313
--webserver_port=34615
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:47:01.844398 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:01.830318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49321 (local address 127.0.97.62:39313)
0929 02:47:01.830502 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:01.830506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:01.831757 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 02:47:01.833705 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:01.833711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:01.833803 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:01.833918 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:01.835650 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.836368 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.837971 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.838381 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.838515 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.844194 (+ 5679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:49321: BlockingRecv error: recv got EOF from 127.0.97.2:49321 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:47:01.845422 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:01.830849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:49321)
0929 02:47:01.831562 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:01.831616 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:01.833420 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:01.834186 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:01.834196 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:01.834894 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:01.835481 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.835498 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.836538 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.836587 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.837822 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.837833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.838599 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.838605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.838847 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.839590 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:01.839611 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:01.844030 (+ 4419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":296,"threads_started":1}
W20260929 02:47:01.846050 9495 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39313 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39313: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:01.854665 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:01.846657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:46703)
0929 02:47:01.846892 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:01.846913 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:01.847062 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:01.847419 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:01.847423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:01.847761 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:01.847899 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.847906 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.848974 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.848979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.849890 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.849900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.850592 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.850598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.850771 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.851385 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:01.851402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:01.854375 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260929 02:47:01.854803 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:01.846730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46703 (local address 127.0.97.62:39313)
0929 02:47:01.846892 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:01.846898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:01.846996 (+ 98us) server_negotiation.cc:407] Connection header received
0929 02:47:01.847188 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:01.847193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:01.847296 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:01.847396 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:01.848033 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.848844 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.850035 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.850448 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.850580 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.854571 (+ 3991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46703: BlockingRecv error: recv got EOF from 127.0.97.2:46703 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:01 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:01.863291 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:01.855529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:46479)
0929 02:47:01.855691 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:01.855709 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:01.855832 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:01.856130 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:01.856134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:01.856441 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:01.856648 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.856658 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.857609 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.857614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.858502 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:01.858512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.859181 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.859186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:01.859366 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.859973 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:01.859989 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:01.862983 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:47:01.863582 9495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:47:01.863693 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:01.855684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46479 (local address 127.0.97.62:39313)
0929 02:47:01.855874 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:01.855881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:01.855900 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:47:01.855957 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:01.855960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:01.856011 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:01.856111 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:01.856781 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.857468 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.858637 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:01.859040 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:01.859172 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:01.863171 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46479: BlockingRecv error: recv got EOF from 127.0.97.2:46479 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 02:47:02.166462 9498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:02.166795 9498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:02.166973 9498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:02.174206 9498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:02.174363 9498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:02.174551 9498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:47:02.184027 9498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:39845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34615
--tserver_master_addrs=127.0.97.62:39313
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:02.185384 9498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:02.186666 9498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:02.199491 9506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:02.199523 9504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:02.199491 9503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:02.200657 9498 server_base.cc:1034] running on GCE node
I20260929 02:47:02.201229 9498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:02.202111 9498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:02.203299 9498 hybrid_clock.cc:648] HybridClock initialized: now 1790650022203262 us; error 38 us; skew 500 ppm
I20260929 02:47:02.203608 9498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650022, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:02.206717 9498 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260929 02:47:02.208143 9498 webserver.cc:460] Webserver started at http://127.0.97.3:34615/ using document root <none> and password file <none>
I20260929 02:47:02.208863 9498 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:02.208971 9498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:02.214139 9498 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:47:02.216977 9513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:02.218349 9498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:47:02.218533 9498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "e99fc0e29ee54b0ba4423fa1b93a7e92"
format_stamp: "Formatted at 2026-09-29 02:46:59 on dist-test-slave-6tjh"
server_key: "561640b51ee22bdc037964f2fbeea473"
server_key_iv: "8c7bd92d7975cd2f4fa6ae8233c8d204"
server_key_version: "encryptionkey@0"
I20260929 02:47:02.219488 9498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:02.240289 9498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:02.244899 9498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:02.245177 9498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:02.246156 9498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:02.247552 9498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:02.247675 9498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:02.247818 9498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:02.247910 9498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:02.280833 9498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39845
I20260929 02:47:02.280916 9625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39845 every 8 connection(s)
I20260929 02:47:02.282310 9498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:47:02.288278 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9498
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 02:47:02.300043 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:02.284871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44871 (local address 127.0.97.62:39313)
0929 02:47:02.285075 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:02.285081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:02.286295 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 02:47:02.288272 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:02.288276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:02.288347 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:02.288485 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:02.290763 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.291672 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.293769 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.294297 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.294433 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.299761 (+ 5328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:44871: BlockingRecv error: recv got EOF from 127.0.97.3:44871 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:47:02.300673 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:02.285419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:44871)
0929 02:47:02.286118 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:02.286162 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:02.287916 (+ 1754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:02.288810 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:02.288823 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:02.289499 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:02.290530 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.290562 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.291870 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.291874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.293577 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.293594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.294574 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.294580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.294867 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.295579 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:02.295638 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:02.299502 (+ 3864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
W20260929 02:47:02.301365 9626 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:39313 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:39313: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260929 02:47:02.306326 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:02.289432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.0.97.62:39313)
0929 02:47:02.291132 (+ 1700us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:02.291136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:02.291150 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:47:02.291188 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:02.291192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:02.291249 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:02.291341 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:02.293003 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.294011 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.295661 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.296336 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.296472 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.299174 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:02.299182 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:02.299201 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:02.299241 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:02.302987 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:02.303842 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:02.303849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:02.303859 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:02.304041 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:02.304428 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:02.304434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:02.304439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:02.305021 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:02.305326 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:02.305657 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 02:47:02.306007 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1614,"thread_start_us":1479,"threads_started":1}
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:02.310505 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:02.301938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:51963)
0929 02:47:02.302110 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:02.302129 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:02.302231 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:02.302526 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:02.302531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:02.302939 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:02.303117 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.303128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.304448 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.304453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.305862 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.305876 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.306632 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.306637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.306831 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.307325 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:02.307350 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:02.310323 (+ 2973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 02:47:02.310966 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:02.302048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51963 (local address 127.0.97.62:39313)
0929 02:47:02.302202 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:02.302207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:02.302222 (+ 15us) server_negotiation.cc:407] Connection header received
0929 02:47:02.302339 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:02.302342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:02.302394 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:02.302471 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:02.303256 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.304291 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.305992 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.306486 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.306611 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.310486 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51963: BlockingRecv error: recv got EOF from 127.0.97.3:51963 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:02.318384 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:02.311140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:45409)
0929 02:47:02.311299 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:02.311316 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:02.311397 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:02.311630 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:02.311632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:02.311882 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:02.312011 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.312017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.313109 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.313113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.314029 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.314040 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.314711 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.314718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.314855 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.315376 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:02.315393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:02.318155 (+ 2762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:47:02.318682 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:02.311184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45409 (local address 127.0.97.62:39313)
0929 02:47:02.311379 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:02.311385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:02.311408 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:47:02.311467 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:02.311470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:02.311524 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:02.311596 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:02.312153 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.312971 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.314157 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.314548 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.314641 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.318381 (+ 3740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45409: BlockingRecv error: recv got EOF from 127.0.97.3:45409 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260929 02:47:02.318724 9626 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:02.398329 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:02.389641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:42571)
0929 02:47:02.390176 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:02.390193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:02.390352 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:02.390687 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:02.390692 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:02.391164 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:02.391355 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.391365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.392342 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.392349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.393381 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.393392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.394331 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.394338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.394532 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.395227 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:02.395245 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:02.397991 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":292,"threads_started":1}
W20260929 02:47:02.398437 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:02.389738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42571 (local address 127.0.97.62:39313)
0929 02:47:02.389990 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:02.389997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:02.390286 (+ 289us) server_negotiation.cc:407] Connection header received
0929 02:47:02.390476 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:02.390481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:02.390553 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:02.390655 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:02.391496 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.392203 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.393525 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.394155 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.394313 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.398198 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42571: BlockingRecv error: recv got EOF from 127.0.97.1:42571 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:02 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:02.872869 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:02.864206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:48613)
0929 02:47:02.864771 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:02.864787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:02.864983 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:02.865457 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:02.865461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:02.866014 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:02.866287 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.866307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.867264 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.867268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.868192 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:02.868202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.868915 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.868920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:02.869056 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.869685 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:02.869703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:02.872637 (+ 2934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":313,"threads_started":1}
W20260929 02:47:02.873051 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:02.864320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48613 (local address 127.0.97.62:39313)
0929 02:47:02.864517 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:02.864524 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:02.864912 (+ 388us) server_negotiation.cc:407] Connection header received
0929 02:47:02.865173 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:02.865202 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:02.865300 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:02.865420 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:02.866442 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.867149 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.868315 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:02.868732 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:02.868821 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:02.872809 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48613: BlockingRecv error: recv got EOF from 127.0.97.2:48613 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 02:47:03 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:03 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:03.332774 9633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:03.319462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:34775)
0929 02:47:03.320007 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:03.320028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:03.320151 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:03.321628 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:03.321633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:03.322352 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:03.322641 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:03.322667 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.324333 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.324338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:03.326088 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:03.326101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.327326 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.327330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:03.327448 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:03.328168 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:03.328439 (+ 271us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:03.332448 (+ 4009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":262,"threads_started":1}
W20260929 02:47:03.332999 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:03.320831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34775 (local address 127.0.97.62:39313)
0929 02:47:03.320995 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:03.321002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:03.321023 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:47:03.321080 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:03.321085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:03.321162 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:03.321329 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:03.322946 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.324024 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.326229 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.327152 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.327304 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:03.332730 (+ 5426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:34775: BlockingRecv error: recv got EOF from 127.0.97.3:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 02:47:03 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:03 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:03.408854 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:03.399473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:48825)
0929 02:47:03.400174 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:03.400192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:03.400331 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:03.400799 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:03.400806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:03.401370 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:03.401551 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:03.401561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.402938 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.402951 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:03.403932 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:03.403943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.404784 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.404790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:03.404972 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:03.405523 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:03.405541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:03.408526 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":432,"threads_started":1}
W20260929 02:47:03.409070 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:03.399573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48825 (local address 127.0.97.62:39313)
0929 02:47:03.399837 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:03.399842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:03.400292 (+ 450us) server_negotiation.cc:407] Connection header received
0929 02:47:03.400453 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:03.400458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:03.400539 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:03.400674 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:03.401699 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.402787 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.404081 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.404622 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.404770 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:03.408813 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48825: BlockingRecv error: recv got EOF from 127.0.97.1:48825 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 02:47:03 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:03 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:03.887586 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:03.873981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:34075)
0929 02:47:03.874538 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:03.874556 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:03.874675 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:03.874966 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:03.874970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:03.875369 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:03.875515 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:03.875524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.876650 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.876656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:03.877578 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:03.877590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.878389 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.878395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:03.878565 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:03.884492 (+ 5927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:03.884512 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:03.887357 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":322,"threads_started":1}
W20260929 02:47:03.887732 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:03.874116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34075 (local address 127.0.97.62:39313)
0929 02:47:03.874317 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:03.874327 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:03.874649 (+ 322us) server_negotiation.cc:407] Connection header received
0929 02:47:03.874766 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:03.874769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:03.874832 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:03.874922 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:03.875682 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.876463 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.877727 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:03.878179 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:03.878344 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:03.887548 (+ 9204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:34075: BlockingRecv error: recv got EOF from 127.0.97.2:34075 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 02:47:04 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:04 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:04.343302 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:04.333809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:47967)
0929 02:47:04.334451 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:04.334470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:04.334636 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:04.334995 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:04.335000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:04.335535 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:04.335691 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:04.335699 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.337028 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.337037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:04.338056 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:04.338068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.338865 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.338872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:04.339056 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:04.339763 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:04.339783 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:04.342923 (+ 3140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":291,"threads_started":1}
W20260929 02:47:04.343442 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:04.334053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47967 (local address 127.0.97.62:39313)
0929 02:47:04.334296 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:04.334306 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:04.334555 (+ 249us) server_negotiation.cc:407] Connection header received
0929 02:47:04.334774 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:04.334781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:04.334857 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:04.334971 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:04.335868 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.336890 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.338217 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.338693 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.338847 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:04.343151 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:47967: BlockingRecv error: recv got EOF from 127.0.97.3:47967 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 29 02:47:04 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:04 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:04.418002 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:04.409846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:41433)
0929 02:47:04.410267 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:04.410283 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:04.410433 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:04.410746 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:04.410750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:04.411124 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:04.411279 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:04.411287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.412282 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.412291 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:04.413234 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:04.413245 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.413930 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.413935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:04.414103 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:04.414684 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:04.414702 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:04.417732 (+ 3030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":219,"threads_started":1}
W20260929 02:47:04.418183 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:04.409871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41433 (local address 127.0.97.62:39313)
0929 02:47:04.410105 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:04.410113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:04.410363 (+ 250us) server_negotiation.cc:407] Connection header received
0929 02:47:04.410564 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:04.410568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:04.410625 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:04.410721 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:04.411416 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.412152 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.413375 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.413787 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.413921 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:04.417914 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:41433: BlockingRecv error: recv got EOF from 127.0.97.1:41433 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 02:47:04 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:04 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:04.896845 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:04.888661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:39817)
0929 02:47:04.889110 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:04.889126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:04.889236 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:04.889494 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:04.889497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:04.889829 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:04.889985 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:04.889992 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.890997 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.891000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:04.891852 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:04.891862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.892669 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.892672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:04.892794 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:04.893398 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:04.893414 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:04.896512 (+ 3098us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":266,"threads_started":1}
W20260929 02:47:04.897002 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:04.888680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39817 (local address 127.0.97.62:39313)
0929 02:47:04.888887 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:04.888897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:04.889206 (+ 309us) server_negotiation.cc:407] Connection header received
0929 02:47:04.889333 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:04.889337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:04.889395 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:04.889479 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:04.890118 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.890878 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.891996 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:04.892468 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:04.892666 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:04.896771 (+ 4105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39817: BlockingRecv error: recv got EOF from 127.0.97.2:39817 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 02:47:05 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:05 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:05.353152 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:05.344386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:36693)
0929 02:47:05.344910 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:05.344925 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:05.345090 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:05.345457 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:05.345462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:05.345824 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:05.345977 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:05.345984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.347029 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.347035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:05.348080 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:05.348091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.348962 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.348968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:05.349151 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:05.349928 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:05.349946 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:05.352890 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":294,"threads_started":1}
W20260929 02:47:05.353319 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:05.344608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36693 (local address 127.0.97.62:39313)
0929 02:47:05.344844 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:05.344851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:05.345016 (+ 165us) server_negotiation.cc:407] Connection header received
0929 02:47:05.345234 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:05.345244 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:05.345315 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:05.345427 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:05.346119 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.346883 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.348223 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.348805 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.348936 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:05.353114 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36693: BlockingRecv error: recv got EOF from 127.0.97.3:36693 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 02:47:05 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:05 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:05.427794 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:05.418988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:36681)
0929 02:47:05.419518 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:05.419536 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:05.419652 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:05.419959 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:05.419962 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:05.420325 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:05.420471 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:05.420480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.421595 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.421601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:05.423013 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:05.423028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.423838 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.423843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:05.423990 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:05.424643 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:05.424670 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:05.427573 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":310,"threads_started":1}
W20260929 02:47:05.428057 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:05.419065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36681 (local address 127.0.97.62:39313)
0929 02:47:05.419285 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:05.419289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:05.419624 (+ 335us) server_negotiation.cc:407] Connection header received
0929 02:47:05.419756 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:05.419759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:05.419828 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:05.419917 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:05.420630 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.421465 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.423177 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.423615 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.423792 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:05.427778 (+ 3986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:36681: BlockingRecv error: recv got EOF from 127.0.97.1:36681 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 29 02:47:05 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:05 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:05.906512 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:05.897870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:55847)
0929 02:47:05.898668 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:05.898693 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:05.898881 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:05.899250 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:05.899255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:05.899634 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:05.899792 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:05.899801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.900841 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.900847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:05.901787 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:05.901798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.902498 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.902504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:05.902676 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:05.903339 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:05.903356 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:05.906187 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":569,"threads_started":1}
W20260929 02:47:05.906621 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:05.898069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55847 (local address 127.0.97.62:39313)
0929 02:47:05.898281 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:05.898287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:05.898817 (+ 530us) server_negotiation.cc:407] Connection header received
0929 02:47:05.899011 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:05.899016 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:05.899104 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:05.899223 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:05.899937 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.900719 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.901936 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:05.902347 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:05.902477 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:05.906393 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55847: BlockingRecv error: recv got EOF from 127.0.97.2:55847 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 02:47:06 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:06 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:06.362603 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:06.354037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:50823)
0929 02:47:06.354512 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:06.354527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:06.354672 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:06.355008 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:06.355013 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:06.355388 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:06.355524 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:06.355533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.356569 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.356573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:06.357532 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:06.357543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.358165 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.358169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:06.358313 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:06.358877 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:06.358893 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:06.362309 (+ 3416us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":256,"threads_started":1}
W20260929 02:47:06.362854 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:06.354143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50823 (local address 127.0.97.62:39313)
0929 02:47:06.354380 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:06.354387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:06.354603 (+ 216us) server_negotiation.cc:407] Connection header received
0929 02:47:06.354800 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:06.354808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:06.354881 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:06.354977 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:06.355677 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.356436 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.357654 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.358046 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.358141 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:06.362560 (+ 4419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50823: BlockingRecv error: recv got EOF from 127.0.97.3:50823 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 02:47:06 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:06 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:06.437590 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:06.428795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:37479)
0929 02:47:06.429314 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:06.429330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:06.429450 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:06.429749 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:06.429753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:06.430159 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:06.430327 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:06.430335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.431357 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.431361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:06.432263 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:06.432273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.433162 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.433171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:06.433380 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:06.434115 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:06.434134 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:06.437350 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":253,"threads_started":1}
W20260929 02:47:06.437704 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:06.428847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37479 (local address 127.0.97.62:39313)
0929 02:47:06.429027 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:06.429032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:06.429412 (+ 380us) server_negotiation.cc:407] Connection header received
0929 02:47:06.429547 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:06.429551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:06.429608 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:06.429694 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:06.430466 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.431204 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.432413 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.432959 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.433091 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:06.437522 (+ 4431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37479: BlockingRecv error: recv got EOF from 127.0.97.1:37479 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 02:47:06 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:06 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:06.915899 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:06.907466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:36201)
0929 02:47:06.908167 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:06.908182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:06.908327 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:06.908667 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:06.908672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:06.909082 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:06.909239 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:06.909248 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.910228 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.910236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:06.911156 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:06.911167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.911853 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.911859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:06.912029 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:06.912653 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:06.912670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:06.915515 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":439,"threads_started":1}
W20260929 02:47:06.915963 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:06.907673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36201 (local address 127.0.97.62:39313)
0929 02:47:06.908013 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:06.908020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:06.908263 (+ 243us) server_negotiation.cc:407] Connection header received
0929 02:47:06.908458 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:06.908462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:06.908518 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:06.908633 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:06.909385 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.910104 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.911291 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:06.911706 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:06.911833 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:06.915718 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36201: BlockingRecv error: recv got EOF from 127.0.97.2:36201 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Sep 29 02:47:07 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:07 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:07.372056 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:07.363538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:56433)
0929 02:47:07.364065 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:07.364082 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:07.364197 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:07.364493 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:07.364496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:07.364917 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:07.365066 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:07.365073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.366090 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.366094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:07.367086 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:07.367099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.367788 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.367794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:07.367965 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:07.368778 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:07.368795 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:07.371716 (+ 2921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":270,"threads_started":1}
W20260929 02:47:07.372176 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:07.363631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56433 (local address 127.0.97.62:39313)
0929 02:47:07.363795 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:07.363799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:07.364157 (+ 358us) server_negotiation.cc:407] Connection header received
0929 02:47:07.364295 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:07.364299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:07.364365 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:07.364454 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:07.365184 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.365934 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.367221 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.367633 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.367774 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:07.371938 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56433: BlockingRecv error: recv got EOF from 127.0.97.3:56433 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:47:07 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:07 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:07.447140 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:07.438705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:54371)
0929 02:47:07.439309 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:07.439325 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:07.439487 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:07.439857 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:07.439861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:07.440291 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:07.440445 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:07.440453 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.441511 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.441519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:07.442436 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:07.442448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.443193 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.443199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:07.443390 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:07.444013 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:07.444031 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:07.446821 (+ 2790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":354,"threads_started":1}
W20260929 02:47:07.447286 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:07.438848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54371 (local address 127.0.97.62:39313)
0929 02:47:07.439107 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:07.439115 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:07.439422 (+ 307us) server_negotiation.cc:407] Connection header received
0929 02:47:07.439615 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:07.439620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:07.439688 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:07.439827 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:07.440618 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.441382 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.442577 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.443032 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.443205 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:07.447036 (+ 3831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54371: BlockingRecv error: recv got EOF from 127.0.97.1:54371 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 29 02:47:07 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:07 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:07.925786 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:07.917014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:51857)
0929 02:47:07.917477 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:07.917492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:07.917639 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:07.917956 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:07.917960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:07.918342 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:07.918515 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:07.918524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.919527 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.919535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:07.920459 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:07.920469 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.921224 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.921230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:07.921405 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:07.921946 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:07.921963 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:07.925506 (+ 3543us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
W20260929 02:47:07.926100 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:07.917029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51857 (local address 127.0.97.62:39313)
0929 02:47:07.917224 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:07.917233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:07.917573 (+ 340us) server_negotiation.cc:407] Connection header received
0929 02:47:07.917763 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:07.917768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:07.917836 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:07.917932 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:07.918659 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.919401 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.920660 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:07.921074 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:07.921207 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:07.925814 (+ 4607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51857: BlockingRecv error: recv got EOF from 127.0.97.2:51857 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:47:08 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:08 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:08.382030 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:08.373156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:40359)
0929 02:47:08.373787 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:08.373804 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:08.373973 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:08.374322 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:08.374326 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:08.374736 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:08.374885 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:08.374893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.376005 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.376009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:08.377185 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:08.377198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.378057 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.378062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:08.378237 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:08.378838 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:08.378857 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:08.381693 (+ 2836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":350,"threads_started":1}
W20260929 02:47:08.382171 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:08.373292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40359 (local address 127.0.97.62:39313)
0929 02:47:08.373563 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:08.373572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:08.373903 (+ 331us) server_negotiation.cc:407] Connection header received
0929 02:47:08.374102 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:08.374107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:08.374175 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:08.374300 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:08.375037 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.375851 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.377336 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.377898 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.378041 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:08.381919 (+ 3878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40359: BlockingRecv error: recv got EOF from 127.0.97.3:40359 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 29 02:47:08 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:08 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:08.457566 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:08.448332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:55967)
0929 02:47:08.448947 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:08.448967 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:08.449109 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:08.449566 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:08.449571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:08.450119 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:08.450404 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:08.450421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.451702 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.451706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:08.452846 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:08.452859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.453490 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.453495 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:08.453609 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:08.454284 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:08.454303 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:08.457306 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":301,"threads_started":1}
W20260929 02:47:08.457659 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:08.448421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55967 (local address 127.0.97.62:39313)
0929 02:47:08.448710 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:08.448719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:08.449075 (+ 356us) server_negotiation.cc:407] Connection header received
0929 02:47:08.449266 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:08.449271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:08.449370 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:08.449493 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:08.450557 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.451567 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.452965 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.453362 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.453455 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:08.457484 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55967: BlockingRecv error: recv got EOF from 127.0.97.1:55967 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:47:08 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:08 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:08.941316 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:08.926848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:41403)
0929 02:47:08.927503 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:08.927520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:08.927677 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:08.928043 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:08.928048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:08.928458 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:08.928670 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:08.928680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.929677 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.929682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:08.930597 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:08.930607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.931583 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.931588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:08.931772 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:08.938230 (+ 6458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:08.938249 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:08.941057 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":267,"threads_started":1}
W20260929 02:47:08.941496 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:08.926949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41403 (local address 127.0.97.62:39313)
0929 02:47:08.927263 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:08.927272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:08.927607 (+ 335us) server_negotiation.cc:407] Connection header received
0929 02:47:08.927854 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:08.927858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:08.927919 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:08.928016 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:08.928808 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.929545 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.930750 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:08.931430 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:08.931558 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:08.941260 (+ 9702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41403: BlockingRecv error: recv got EOF from 127.0.97.2:41403 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Sep 29 02:47:09 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:09 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:09.391130 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:09.383051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:37253)
0929 02:47:09.383592 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:09.383608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:09.383721 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:09.384012 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:09.384015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:09.384384 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:09.384578 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:09.384586 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.385626 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.385630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:09.386554 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:09.386564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.387330 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.387336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:09.387454 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:09.388015 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:09.388032 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:09.390890 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":263,"threads_started":1}
W20260929 02:47:09.391429 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:09.383145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37253 (local address 127.0.97.62:39313)
0929 02:47:09.383381 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:09.383391 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:09.383689 (+ 298us) server_negotiation.cc:407] Connection header received
0929 02:47:09.383830 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:09.383833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:09.383895 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:09.383974 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:09.384775 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.385500 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.386675 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.387148 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.387283 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:09.391123 (+ 3840us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:37253: BlockingRecv error: recv got EOF from 127.0.97.3:37253 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 02:47:09 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:09 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:09.466655 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:09.458546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:45437)
0929 02:47:09.459029 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:09.459045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:09.459153 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:09.459492 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:09.459495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:09.459905 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:09.460094 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:09.460108 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.461199 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.461205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:09.462140 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:09.462151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.462811 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.462816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:09.462927 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:09.463481 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:09.463498 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:09.466322 (+ 2824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":277,"threads_started":1}
W20260929 02:47:09.466778 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:09.458735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45437 (local address 127.0.97.62:39313)
0929 02:47:09.458985 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:09.458993 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:09.459112 (+ 119us) server_negotiation.cc:407] Connection header received
0929 02:47:09.459260 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:09.459263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:09.459379 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:09.459455 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:09.460212 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.461044 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.462265 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.462656 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.462745 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:09.466505 (+ 3760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:45437: BlockingRecv error: recv got EOF from 127.0.97.1:45437 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 29 02:47:09 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:09 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:09.951346 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:09.942300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:54629)
0929 02:47:09.942826 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:09.942842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:09.942961 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:09.943308 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:09.943311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:09.943763 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:09.943952 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:09.943961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.945544 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.945549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:09.946511 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:09.946522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.947410 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.947415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:09.947531 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:09.948129 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:09.948146 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:09.951009 (+ 2863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":289,"threads_started":1}
W20260929 02:47:09.951406 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:09.942417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54629 (local address 127.0.97.62:39313)
0929 02:47:09.942611 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:09.942616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:09.942940 (+ 324us) server_negotiation.cc:407] Connection header received
0929 02:47:09.943112 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:09.943115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:09.943179 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:09.943272 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:09.944092 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.945398 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.946655 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:09.947245 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:09.947345 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:09.951195 (+ 3850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54629: BlockingRecv error: recv got EOF from 127.0.97.2:54629 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:47:10 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:10 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:10.400877 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:10.392106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:50265)
0929 02:47:10.392807 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:10.392825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:10.392986 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:10.393359 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:10.393363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:10.393838 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:10.394040 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:10.394049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.395182 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.395186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:10.396147 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:10.396159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.396954 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.396959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:10.397097 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:10.397771 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:10.397792 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:10.400596 (+ 2804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":462,"threads_started":1}
W20260929 02:47:10.401023 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:10.392161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50265 (local address 127.0.97.62:39313)
0929 02:47:10.392455 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:10.392464 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:10.392910 (+ 446us) server_negotiation.cc:407] Connection header received
0929 02:47:10.393114 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:10.393120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:10.393198 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:10.393336 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:10.394164 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.395043 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.396295 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.396758 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.396864 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:10.400764 (+ 3900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:50265: BlockingRecv error: recv got EOF from 127.0.97.3:50265 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 02:47:10 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:10 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:10.475697 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:10.467659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:42947)
0929 02:47:10.468150 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:10.468168 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:10.468286 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:10.468763 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:10.468767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:10.469140 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:10.469318 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:10.469326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.470300 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.470304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:10.471270 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:10.471281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.471944 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.471948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:10.472069 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:10.472740 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:10.472758 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:10.475452 (+ 2694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":263,"threads_started":1}
W20260929 02:47:10.476011 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:10.467765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42947 (local address 127.0.97.62:39313)
0929 02:47:10.468003 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:10.468010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:10.468286 (+ 276us) server_negotiation.cc:407] Connection header received
0929 02:47:10.468457 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:10.468461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:10.468611 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:10.468696 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:10.469449 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.470168 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.471407 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.471817 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.471912 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:10.475740 (+ 3828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42947: BlockingRecv error: recv got EOF from 127.0.97.1:42947 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 02:47:10 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:10 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:10.961186 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:10.952375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:53233)
0929 02:47:10.952891 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:10.952907 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:10.953053 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:10.953370 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:10.953374 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:10.953795 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:10.953955 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:10.953963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.954957 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.954965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:10.956120 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:10.956136 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.957176 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.957180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:10.957320 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:10.957976 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:10.957995 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:10.960823 (+ 2828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":292,"threads_started":1}
W20260929 02:47:10.961493 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:10.952382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53233 (local address 127.0.97.62:39313)
0929 02:47:10.952601 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:10.952609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:10.952987 (+ 378us) server_negotiation.cc:407] Connection header received
0929 02:47:10.953176 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:10.953180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:10.953240 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:10.953345 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:10.954099 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.954830 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.956315 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:10.957048 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:10.957153 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:10.961124 (+ 3971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:53233: BlockingRecv error: recv got EOF from 127.0.97.2:53233 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 02:47:11 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:11 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:11.410293 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:11.401814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:38407)
0929 02:47:11.402425 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:11.402443 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:11.402566 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:11.402941 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:11.402945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:11.403369 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:11.403536 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:11.403545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.404581 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.404585 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:11.405544 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:11.405555 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.406342 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.406346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:11.406467 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:11.407122 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:11.407140 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:11.410049 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":349,"threads_started":1}
W20260929 02:47:11.410481 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:11.401849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38407 (local address 127.0.97.62:39313)
0929 02:47:11.402016 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:11.402021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:11.402543 (+ 522us) server_negotiation.cc:407] Connection header received
0929 02:47:11.402707 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:11.402710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:11.402782 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:11.402872 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:11.403677 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.404424 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.405706 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.406203 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.406315 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:11.410271 (+ 3956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38407: BlockingRecv error: recv got EOF from 127.0.97.3:38407 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 02:47:11 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:11 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:11.485139 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:11.476764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:50667)
0929 02:47:11.477239 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:11.477256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:11.477365 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:11.477673 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:11.477676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:11.478052 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:11.478190 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:11.478197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.479322 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.479327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:11.480253 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:11.480266 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.481115 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.481119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:11.481242 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:11.481868 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:11.481885 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:11.484835 (+ 2950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":269,"threads_started":1}
W20260929 02:47:11.485352 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:11.476836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50667 (local address 127.0.97.62:39313)
0929 02:47:11.477081 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:11.477089 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:11.477338 (+ 249us) server_negotiation.cc:407] Connection header received
0929 02:47:11.477463 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:11.477467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:11.477547 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:11.477636 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:11.478320 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.479174 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.480393 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.480959 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.481059 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:11.485038 (+ 3979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:50667: BlockingRecv error: recv got EOF from 127.0.97.1:50667 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 02:47:11 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:11 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:11.970780 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:11.962279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:36887)
0929 02:47:11.962835 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:11.962851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:11.962957 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:11.963228 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:11.963231 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:11.963603 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:11.963768 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:11.963777 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.964729 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.964733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:11.965654 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:11.965664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.966470 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.966476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:11.966631 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:11.967418 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:11.967440 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:11.970565 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
W20260929 02:47:11.971201 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:11.962345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36887 (local address 127.0.97.62:39313)
0929 02:47:11.962528 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:11.962533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:11.962920 (+ 387us) server_negotiation.cc:407] Connection header received
0929 02:47:11.963047 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:11.963050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:11.963108 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:11.963189 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:11.963885 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.964591 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.965821 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:11.966296 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:11.966399 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:11.970900 (+ 4501us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:36887: BlockingRecv error: recv got EOF from 127.0.97.2:36887 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:47:12 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:12 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:12.420152 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:12.411367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:58705)
0929 02:47:12.412020 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:12.412037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:12.412197 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:12.412605 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:12.412610 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:12.413033 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:12.413179 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:12.413187 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.414259 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.414268 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:12.415308 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:12.415319 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.416071 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.416078 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:12.416268 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:12.416966 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:12.416986 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:12.419801 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":417,"threads_started":1}
W20260929 02:47:12.420279 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:12.411490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58705 (local address 127.0.97.62:39313)
0929 02:47:12.411667 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:12.411675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:12.412130 (+ 455us) server_negotiation.cc:407] Connection header received
0929 02:47:12.412339 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:12.412344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:12.412431 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:12.412539 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:12.413332 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.414118 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.415461 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.415911 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.416053 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:12.419986 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58705: BlockingRecv error: recv got EOF from 127.0.97.3:58705 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 29 02:47:12 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:12 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:12.494342 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:12.486064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:51733)
0929 02:47:12.486563 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:12.486579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:12.486730 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:12.487087 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:12.487092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:12.487509 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:12.487671 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:12.487680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.488732 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.488739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:12.489651 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:12.489663 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.490401 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.490407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:12.490576 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:12.491154 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:12.491171 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:12.493992 (+ 2821us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":272,"threads_started":1}
W20260929 02:47:12.494499 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:12.486149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51733 (local address 127.0.97.62:39313)
0929 02:47:12.486408 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:12.486414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:12.486659 (+ 245us) server_negotiation.cc:407] Connection header received
0929 02:47:12.486872 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:12.486878 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:12.486960 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:12.487057 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:12.487812 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.488605 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.489787 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.490244 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.490372 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:12.494189 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51733: BlockingRecv error: recv got EOF from 127.0.97.1:51733 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 02:47:12 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:12 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:12.979966 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:12.971692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:44577)
0929 02:47:12.972213 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:12.972228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:12.972381 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:12.972756 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:12.972761 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:12.973188 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:12.973367 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:12.973375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.974353 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.974359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:12.975256 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:12.975266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.975946 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.975952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:12.976123 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:12.976870 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:12.976887 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:12.979727 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":279,"threads_started":1}
W20260929 02:47:12.980160 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:12.971819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44577 (local address 127.0.97.62:39313)
0929 02:47:12.972049 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:12.972056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:12.972311 (+ 255us) server_negotiation.cc:407] Connection header received
0929 02:47:12.972514 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:12.972519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:12.972598 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:12.972695 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:12.973504 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.974222 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.975386 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:12.975807 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:12.975934 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:12.979932 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44577: BlockingRecv error: recv got EOF from 127.0.97.2:44577 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 29 02:47:13 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:13 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:13.429589 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:13.421265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:51563)
0929 02:47:13.421827 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:13.421852 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:13.421997 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:13.422320 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:13.422324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:13.422802 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:13.422983 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:13.422991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.424008 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.424015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:13.424969 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:13.424980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.425728 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.425734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:13.425907 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:13.426460 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:13.426476 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:13.429232 (+ 2756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":295,"threads_started":1}
W20260929 02:47:13.429669 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:13.421351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51563 (local address 127.0.97.62:39313)
0929 02:47:13.421544 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:13.421553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:13.421929 (+ 376us) server_negotiation.cc:407] Connection header received
0929 02:47:13.422129 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:13.422133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:13.422192 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:13.422294 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:13.423115 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.423894 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.425102 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.425577 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.425701 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:13.429449 (+ 3748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:51563: BlockingRecv error: recv got EOF from 127.0.97.3:51563 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 02:47:13 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:13 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:13.503404 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:13.495388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:55597)
0929 02:47:13.495931 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:13.495947 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:13.496055 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:13.496319 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:13.496322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:13.496749 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:13.496907 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:13.496916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.497879 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.497884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:13.498854 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:13.498866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.499556 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.499560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:13.499686 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:13.500363 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:13.500380 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:13.503164 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":305,"threads_started":1}
W20260929 02:47:13.503557 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:13.495434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55597 (local address 127.0.97.62:39313)
0929 02:47:13.495623 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:13.495628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:13.496021 (+ 393us) server_negotiation.cc:407] Connection header received
0929 02:47:13.496149 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:13.496153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:13.496226 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:13.496308 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:13.497030 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.497738 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.498975 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.499434 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.499524 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:13.503338 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:55597: BlockingRecv error: recv got EOF from 127.0.97.1:55597 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 29 02:47:13 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:13 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:13.992942 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:13.980953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:44129)
0929 02:47:13.981534 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:13.981550 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:13.981662 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:13.982006 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:13.982010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:13.982396 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:13.982555 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:13.982563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.983547 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.983552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:13.984481 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:13.984493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.985166 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.985171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:13.985308 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:13.989556 (+ 4248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:13.989575 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:13.992680 (+ 3105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":330,"threads_started":1}
W20260929 02:47:13.993052 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:13.980988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44129 (local address 127.0.97.62:39313)
0929 02:47:13.981158 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:13.981163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:13.981621 (+ 458us) server_negotiation.cc:407] Connection header received
0929 02:47:13.981773 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:13.981777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:13.981837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:13.981924 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:13.982674 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.983403 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.984641 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:13.985046 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:13.985140 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:13.992852 (+ 7712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:44129: BlockingRecv error: recv got EOF from 127.0.97.2:44129 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 02:47:14 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:14 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:14.439324 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:14.430544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:46445)
0929 02:47:14.431180 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:14.431199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:14.431327 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:14.431742 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:14.431746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:14.432130 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:14.432290 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:14.432299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.433378 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.433382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:14.434293 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:14.434304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.435047 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.435051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:14.435183 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:14.435938 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:14.435962 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:14.439089 (+ 3127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":375,"threads_started":1}
W20260929 02:47:14.439539 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:14.430567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46445 (local address 127.0.97.62:39313)
0929 02:47:14.430727 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:14.430732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:14.431284 (+ 552us) server_negotiation.cc:407] Connection header received
0929 02:47:14.431428 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:14.431432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:14.431496 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:14.431584 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:14.432422 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.433234 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.434411 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.434918 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.435013 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:14.439313 (+ 4300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:46445: BlockingRecv error: recv got EOF from 127.0.97.3:46445 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 02:47:14 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:14 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:14.512881 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:14.504334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:56139)
0929 02:47:14.504849 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:14.504865 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:14.504977 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:14.505299 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:14.505303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:14.505670 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:14.505849 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:14.505857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.506885 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.506888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:14.507790 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:14.507800 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.508706 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.508711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:14.508948 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:14.509597 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:14.509614 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:14.512452 (+ 2838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
W20260929 02:47:14.512948 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:14.504443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56139 (local address 127.0.97.62:39313)
0929 02:47:14.504702 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:14.504710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:14.504941 (+ 231us) server_negotiation.cc:407] Connection header received
0929 02:47:14.505073 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:14.505076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:14.505156 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:14.505262 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:14.505962 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.506763 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.507937 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.508495 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.508675 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:14.512699 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56139: BlockingRecv error: recv got EOF from 127.0.97.1:56139 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Sep 29 02:47:14 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:14 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:15.002050 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:14.993992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:41039)
0929 02:47:14.994564 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:14.994581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:14.994761 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:14.995138 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:14.995142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:14.995583 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:14.995751 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:14.995760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.996728 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.996732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:14.997633 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:14.997642 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.998332 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.998336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:14.998447 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:14.999013 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:14.999028 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:15.001830 (+ 2802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":307,"threads_started":1}
W20260929 02:47:15.002224 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:14.994175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41039 (local address 127.0.97.62:39313)
0929 02:47:14.994388 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:14.994394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:14.994665 (+ 271us) server_negotiation.cc:407] Connection header received
0929 02:47:14.994869 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:14.994872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:14.994932 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:14.995028 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:14.995883 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.996624 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.997746 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:14.998181 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:14.998272 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:15.001980 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41039: BlockingRecv error: recv got EOF from 127.0.97.2:41039 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 29 02:47:15 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:15 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:15.449365 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:15.440355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:60333)
0929 02:47:15.440907 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:15.440924 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:15.441042 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:15.441361 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:15.441364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:15.441787 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:15.441937 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:15.441946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.443000 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.443006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:15.444293 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:15.444312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.445358 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.445363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:15.445483 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:15.446136 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:15.446153 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:15.449021 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
W20260929 02:47:15.449445 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:15.440444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60333 (local address 127.0.97.62:39313)
0929 02:47:15.440653 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:15.440658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:15.441031 (+ 373us) server_negotiation.cc:407] Connection header received
0929 02:47:15.441155 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:15.441159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:15.441225 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:15.441320 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:15.442081 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.442818 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.444446 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.445151 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.445272 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:15.449212 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60333: BlockingRecv error: recv got EOF from 127.0.97.3:60333 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 29 02:47:15 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:15 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:15.522657 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:15.513915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:44301)
0929 02:47:15.514420 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:15.514436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:15.514587 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:15.514945 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:15.514950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:15.515342 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:15.515507 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:15.515516 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.516855 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.516861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:15.517763 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:15.517776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.518480 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.518486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:15.518659 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:15.519310 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:15.519329 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:15.522266 (+ 2937us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":268,"threads_started":1}
W20260929 02:47:15.522727 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:15.514023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44301 (local address 127.0.97.62:39313)
0929 02:47:15.514327 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:15.514333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:15.514514 (+ 181us) server_negotiation.cc:407] Connection header received
0929 02:47:15.514715 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:15.514719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:15.514816 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:15.514918 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:15.515670 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.516724 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.517914 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:15.518316 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:15.518469 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:15.522470 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:44301: BlockingRecv error: recv got EOF from 127.0.97.1:44301 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
Sep 29 02:47:16 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:16 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:16.011396 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:16.003016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:57075)
0929 02:47:16.003484 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:16.003499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:16.003651 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:16.003974 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:16.003978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:16.004371 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:16.004574 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:16.004583 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.005596 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.005602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:16.006514 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:16.006525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.007226 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.007231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:16.007410 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:16.008052 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:16.008070 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:16.011041 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":253,"threads_started":1}
W20260929 02:47:16.011461 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:16.003095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57075 (local address 127.0.97.62:39313)
0929 02:47:16.003328 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:16.003336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:16.003584 (+ 248us) server_negotiation.cc:407] Connection header received
0929 02:47:16.003769 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:16.003774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:16.003844 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:16.003943 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:16.004730 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.005459 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.006651 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.007074 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.007201 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:16.011238 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:57075: BlockingRecv error: recv got EOF from 127.0.97.2:57075 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 29 02:47:16 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:16 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:16.459110 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:16.450322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.3:60359)
0929 02:47:16.450874 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:16.450892 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:16.451023 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:16.451353 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:16.451357 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:16.451785 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:16.452023 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:16.452036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.453366 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.453373 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:16.454494 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:16.454507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.455199 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.455204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:16.455387 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:16.456040 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:16.456057 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:16.458870 (+ 2813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":310,"threads_started":1}
W20260929 02:47:16.459326 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:16.450437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60359 (local address 127.0.97.62:39313)
0929 02:47:16.450607 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:16.450612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:16.450979 (+ 367us) server_negotiation.cc:407] Connection header received
0929 02:47:16.451128 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:16.451132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:16.451206 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:16.451303 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:16.452190 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.453210 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.454638 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.455052 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.455186 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:16.459075 (+ 3889us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60359: BlockingRecv error: recv got EOF from 127.0.97.3:60359 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 02:47:16 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:16 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:16.532976 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:16.524028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.1:48547)
0929 02:47:16.524686 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:16.524727 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:16.524887 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:16.525301 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:16.525306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:16.525788 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:16.525977 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:16.525985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.527129 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.527135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:16.528064 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:16.528075 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.528940 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.528947 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:16.529126 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:16.529794 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:16.529812 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:16.532583 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":337,"threads_started":1}
W20260929 02:47:16.533049 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:16.524221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48547 (local address 127.0.97.62:39313)
0929 02:47:16.524463 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:16.524469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:16.524819 (+ 350us) server_negotiation.cc:407] Connection header received
0929 02:47:16.525038 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:16.525042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:16.525151 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:16.525273 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:16.526130 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.526991 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.528211 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:16.528791 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:16.528929 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:16.532822 (+ 3893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48547: BlockingRecv error: recv got EOF from 127.0.97.1:48547 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 02:47:17 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 02:47:17 dist-test-slave-6tjh krb5kdc[8628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 02:47:17.021093 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:17.012624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39313 (local address 127.0.97.2:59473)
0929 02:47:17.013182 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:17.013198 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:17.013369 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:17.013723 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:17.013728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:17.014145 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:17.014366 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:17.014375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.015424 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.015429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:17.016329 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:17.016339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.017136 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.017142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:17.017304 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:17.017893 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:17.017910 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:17.020716 (+ 2806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:39313: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":300,"threads_started":1}
W20260929 02:47:17.021150 9365 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:17.012818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59473 (local address 127.0.97.62:39313)
0929 02:47:17.013052 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:17.013058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:17.013294 (+ 236us) server_negotiation.cc:407] Connection header received
0929 02:47:17.013507 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:17.013511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:17.013580 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:17.013679 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:17.014503 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.015277 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.016462 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.016990 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.017125 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:17.020914 (+ 3789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:59473: BlockingRecv error: recv got EOF from 127.0.97.2:59473 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:47:17.297179 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9216
I20260929 02:47:17.309615 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:17.298239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.0.97.1:41427)
0929 02:47:17.298796 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:17.298804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:17.298830 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:47:17.298903 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:17.298909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:17.299125 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:17.299303 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:17.300797 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.301809 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.303044 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.303605 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.303714 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:17.306462 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:17.306470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:17.306484 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:17.306516 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:17.309125 (+ 2609us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:47:17.309379 (+ 254us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40934: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":293,"threads_started":1}
W20260929 02:47:17.309746 9676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40934: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:47:17.309800 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9216: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:41427: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:47:17.310051 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9216
I20260929 02:47:17.326355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9367
I20260929 02:47:17.339836 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:17.327502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47484 (local address 127.0.97.2:36197)
0929 02:47:17.328090 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:17.328097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:17.328128 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:47:17.328211 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:17.328218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:17.328445 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:17.328702 (+ 257us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:17.330126 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.331236 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.332679 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.333284 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.333393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:17.336190 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:17.336198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:17.336215 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:17.336253 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:17.339117 (+ 2864us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:47:17.339541 (+ 424us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47484: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":296,"threads_started":1}
W20260929 02:47:17.339946 9677 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47484: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:47:17.340021 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9367: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:36197: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:47:17.340194 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9367
I20260929 02:47:17.357242 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9498
I20260929 02:47:17.369778 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:17.358354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.0.97.3:39845)
0929 02:47:17.358962 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:17.358969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:17.358997 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:47:17.359057 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:17.359063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:17.359280 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:17.359464 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:17.360856 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.361935 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.363157 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:17.363739 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:17.363883 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:17.366658 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:17.366668 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:17.366683 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:17.366723 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:17.369261 (+ 2538us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 02:47:17.369538 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42592: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":335,"threads_started":1}
W20260929 02:47:17.369906 9678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42592: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 02:47:17.370040 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9498: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:39845: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 02:47:17.370216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9498
I20260929 02:47:17.386373 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9147
I20260929 02:47:17.387607 9209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:17.504618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9147
2026-09-29T02:47:17Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21184 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T02:47:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:47:17Z Disabled control of system clock
I20260929 02:47:17.583221 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34735
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34735
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 02:47:17.917346 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:17.917654 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:17.917824 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:17.925163 9683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:47:17.925274 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:17.925343 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:17.925448 9683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:47:17.925535 9683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:47:17.935034 9683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:17.936424 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:17.938336 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:17.951155 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:17.951227 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:17.951249 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:17.951709 9683 server_base.cc:1034] running on GCE node
I20260929 02:47:17.952621 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:17.954088 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:17.955327 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790650037955275 us; error 52 us; skew 500 ppm
I20260929 02:47:17.955708 9683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:17.958153 9683 webserver.cc:460] Webserver started at http://127.0.97.62:44881/ using document root <none> and password file <none>
I20260929 02:47:17.958899 9683 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:17.959033 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:17.959362 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:17.962386 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf"
format_stamp: "Formatted at 2026-09-29 02:47:17 on dist-test-slave-6tjh"
server_key: "7f3a050358aa9c0d5c04b8075d3656fb"
server_key_iv: "f2d3fdc589356f9365331def56544c75"
server_key_version: "encryptionkey@0"
I20260929 02:47:17.963233 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf"
format_stamp: "Formatted at 2026-09-29 02:47:17 on dist-test-slave-6tjh"
server_key: "7f3a050358aa9c0d5c04b8075d3656fb"
server_key_iv: "f2d3fdc589356f9365331def56544c75"
server_key_version: "encryptionkey@0"
I20260929 02:47:17.968501 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:47:17.972097 9697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:17.973634 9683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:47:17.973831 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf"
format_stamp: "Formatted at 2026-09-29 02:47:17 on dist-test-slave-6tjh"
server_key: "7f3a050358aa9c0d5c04b8075d3656fb"
server_key_iv: "f2d3fdc589356f9365331def56544c75"
server_key_version: "encryptionkey@0"
I20260929 02:47:17.974058 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:18.002537 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:18.003427 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:18.003731 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:18.021809 9683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34735
I20260929 02:47:18.021812 9748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34735 every 8 connection(s)
I20260929 02:47:18.023185 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:47:18.027019 9749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:47:18.027087 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9683
I20260929 02:47:18.027254 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:47:18.028079 388 external_mini_cluster.cc:1428] Setting key 55102f297280b627762e922d771c7cd1
I20260929 02:47:18.039490 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf: Bootstrap starting.
I20260929 02:47:18.042119 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:18.030266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.0.97.62:34735)
0929 02:47:18.031271 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:18.031289 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:18.031340 (+ 51us) server_negotiation.cc:407] Connection header received
0929 02:47:18.033492 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:18.033515 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:18.034010 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:18.034424 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:18.035045 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:18.036312 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:18.037683 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:18.038363 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:18.038574 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:18.039818 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:18.039825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:18.039839 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:18.040377 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:18.040452 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:18.040473 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:18.040625 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:18.040883 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 02:47:18.041179 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":310,"threads_started":1}
I20260929 02:47:18.043015 9749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf: Neither blocks nor log segments found. Creating new log.
I20260929 02:47:18.044188 9749 log.cc:824] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf: Log is configured to *not* fsync() on all Append() calls
I20260929 02:47:18.047295 9749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf: No bootstrap required, opened a new log
I20260929 02:47:18.054839 9749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34735 } }
I20260929 02:47:18.055245 9749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:47:18.055392 9749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f8cd4a2cc6a4e13996e6b690db8afcf, State: Initialized, Role: FOLLOWER
I20260929 02:47:18.055974 9749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34735 } }
I20260929 02:47:18.056210 9749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:47:18.056356 9749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:47:18.056523 9749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:47:18.059003 9749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34735 } }
I20260929 02:47:18.059500 9749 leader_election.cc:302] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f8cd4a2cc6a4e13996e6b690db8afcf; no voters:
I20260929 02:47:18.060091 9749 leader_election.cc:288] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:47:18.060302 9754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:47:18.061105 9754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [term 1 LEADER]: Becoming Leader. State: Replica: 5f8cd4a2cc6a4e13996e6b690db8afcf, State: Running, Role: LEADER
I20260929 02:47:18.061671 9754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34735 } }
I20260929 02:47:18.062273 9749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:47:18.064370 9756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f8cd4a2cc6a4e13996e6b690db8afcf. Latest consensus state: current_term: 1 leader_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34735 } } }
I20260929 02:47:18.064808 9756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:18.065765 9755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f8cd4a2cc6a4e13996e6b690db8afcf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34735 } } }
I20260929 02:47:18.066061 9755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:18.070083 9763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:47:18.075381 9763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:47:18.083604 9763 catalog_manager.cc:1347] Generated new cluster ID: bd8704a10f6f4557beaac92b381196bf
I20260929 02:47:18.083719 9763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:47:18.098573 9763 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:47:18.100449 9763 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:47:18.109331 9763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f8cd4a2cc6a4e13996e6b690db8afcf: Generated new TSK 0
I20260929 02:47:18.110466 9763 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:47:18.118809 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34735
--builtin_ntp_servers=127.0.97.20:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 02:47:18.452659 9773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:18.452960 9773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:18.453045 9773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:18.460183 9773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:18.460301 9773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:18.460408 9773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:47:18.470003 9773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:18.471298 9773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:18.473178 9773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:18.486505 9778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:18.486546 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:18.486569 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:18.487465 9773 server_base.cc:1034] running on GCE node
I20260929 02:47:18.488062 9773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:18.489051 9773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:18.490248 9773 hybrid_clock.cc:648] HybridClock initialized: now 1790650038490217 us; error 37 us; skew 500 ppm
I20260929 02:47:18.490530 9773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:18.492679 9773 webserver.cc:460] Webserver started at http://127.0.97.1:40137/ using document root <none> and password file <none>
I20260929 02:47:18.493319 9773 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:18.493404 9773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:18.493664 9773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:18.496342 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "fc9e8e48dc894112884de0c1ce33d487"
format_stamp: "Formatted at 2026-09-29 02:47:18 on dist-test-slave-6tjh"
server_key: "a23326570f9e2dc5a0b4f54b7e990bfa"
server_key_iv: "719f2c0b948cf2b2011691c61fbd1088"
server_key_version: "encryptionkey@0"
I20260929 02:47:18.497138 9773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fc9e8e48dc894112884de0c1ce33d487"
format_stamp: "Formatted at 2026-09-29 02:47:18 on dist-test-slave-6tjh"
server_key: "a23326570f9e2dc5a0b4f54b7e990bfa"
server_key_iv: "719f2c0b948cf2b2011691c61fbd1088"
server_key_version: "encryptionkey@0"
I20260929 02:47:18.502760 9773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:47:18.506117 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:18.507619 9773 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:47:18.507798 9773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "fc9e8e48dc894112884de0c1ce33d487"
format_stamp: "Formatted at 2026-09-29 02:47:18 on dist-test-slave-6tjh"
server_key: "a23326570f9e2dc5a0b4f54b7e990bfa"
server_key_iv: "719f2c0b948cf2b2011691c61fbd1088"
server_key_version: "encryptionkey@0"
I20260929 02:47:18.508056 9773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:18.544796 9773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:18.545765 9773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:18.546162 9773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:18.547156 9773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:18.548735 9773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:18.548838 9773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:18.548924 9773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:18.548983 9773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:18.585326 9773 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44093
I20260929 02:47:18.585419 9899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44093 every 8 connection(s)
I20260929 02:47:18.586740 9773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:47:18.591766 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9773
I20260929 02:47:18.591992 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:47:18.592729 388 external_mini_cluster.cc:1428] Setting key 88190c7d25b407ef8a9edf6154b321d0
I20260929 02:47:18.595460 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34735
--builtin_ntp_servers=127.0.97.20:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:47:18.599179 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:18.589325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48267 (local address 127.0.97.62:34735)
0929 02:47:18.589775 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:18.589780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:18.590908 (+ 1128us) server_negotiation.cc:407] Connection header received
0929 02:47:18.592791 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:18.592797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:18.592888 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:18.593003 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:18.594042 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:18.594766 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:18.596402 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:18.596875 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:18.597004 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:18.598385 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:18.598390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:18.598393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:18.598409 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:18.598443 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:18.598454 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:18.598547 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:18.598767 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 02:47:18.598860 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":254,"threads_started":1}
I20260929 02:47:18.600214 9903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:18.589855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34735 (local address 127.0.97.1:48267)
0929 02:47:18.590731 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:18.590765 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:18.592471 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:18.593235 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:18.593243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:18.593331 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:18.593882 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:18.593896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:18.594924 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:18.594929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:18.596250 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:18.596261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:18.597084 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:18.597093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:18.597327 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:18.598061 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:47:18.598063 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:18.598109 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:47:18.598188 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:18.598546 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:18.598557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:18.598567 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:47:18.598751 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 02:47:18.599156 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":276,"threads_started":1}
I20260929 02:47:18.601773 9900 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34735
I20260929 02:47:18.602079 9900 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:18.602806 9900 heartbeater.cc:511] Master 127.0.97.62:34735 requested a full tablet report, sending...
I20260929 02:47:18.605069 9714 ts_manager.cc:194] Registered new tserver with Master: fc9e8e48dc894112884de0c1ce33d487 (127.0.97.1:44093)
I20260929 02:47:18.607437 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48267
W20260929 02:47:18.934932 9904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:18.935240 9904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:18.935324 9904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:18.942832 9904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:18.942958 9904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:18.943151 9904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:47:18.952922 9904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:18.954252 9904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:18.956033 9904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:18.969916 9909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:18.969959 9910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:18.970352 9912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:18.970726 9904 server_base.cc:1034] running on GCE node
I20260929 02:47:18.971328 9904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:18.972240 9904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:18.973441 9904 hybrid_clock.cc:648] HybridClock initialized: now 1790650038973358 us; error 78 us; skew 500 ppm
I20260929 02:47:18.973778 9904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:18.976228 9904 webserver.cc:460] Webserver started at http://127.0.97.2:37143/ using document root <none> and password file <none>
I20260929 02:47:18.977020 9904 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:18.977113 9904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:18.977473 9904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:18.980645 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "9264cb511f8e41088d409976467d4d7c"
format_stamp: "Formatted at 2026-09-29 02:47:18 on dist-test-slave-6tjh"
server_key: "0e76176ffb4acbd85afa0f48c8f54e3d"
server_key_iv: "84114925bd7b8b166881260db6b666b7"
server_key_version: "encryptionkey@0"
I20260929 02:47:18.981485 9904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9264cb511f8e41088d409976467d4d7c"
format_stamp: "Formatted at 2026-09-29 02:47:18 on dist-test-slave-6tjh"
server_key: "0e76176ffb4acbd85afa0f48c8f54e3d"
server_key_iv: "84114925bd7b8b166881260db6b666b7"
server_key_version: "encryptionkey@0"
I20260929 02:47:18.986900 9904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:47:18.990569 9918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:18.992022 9904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:47:18.992206 9904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "9264cb511f8e41088d409976467d4d7c"
format_stamp: "Formatted at 2026-09-29 02:47:18 on dist-test-slave-6tjh"
server_key: "0e76176ffb4acbd85afa0f48c8f54e3d"
server_key_iv: "84114925bd7b8b166881260db6b666b7"
server_key_version: "encryptionkey@0"
I20260929 02:47:18.992425 9904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:19.016978 9904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:19.017889 9904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:19.018175 9904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:19.019156 9904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:19.020695 9904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:19.020814 9904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:19.020933 9904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:19.021004 9904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:19.054929 9904 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37313
I20260929 02:47:19.054982 10030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37313 every 8 connection(s)
I20260929 02:47:19.056272 9904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:47:19.062014 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 9904
I20260929 02:47:19.062178 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:47:19.062698 388 external_mini_cluster.cc:1428] Setting key 245c3d45d160e1f270d02562e2df6417
I20260929 02:47:19.065810 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34735
--builtin_ntp_servers=127.0.97.20:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 02:47:19.070060 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:19.058990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43825 (local address 127.0.97.62:34735)
0929 02:47:19.059157 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:19.059162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:19.060686 (+ 1524us) server_negotiation.cc:407] Connection header received
0929 02:47:19.062998 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:19.063003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:19.063125 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:19.063220 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:19.064619 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.065489 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.067260 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.067646 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.067792 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:19.069222 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:19.069226 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:19.069230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:19.069248 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:19.069276 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:19.069288 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:19.069381 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:19.069611 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 02:47:19.069748 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:47:19.071077 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:19.059547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34735 (local address 127.0.97.2:43825)
0929 02:47:19.060477 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:19.060512 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:19.062626 (+ 2114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:19.063465 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:19.063474 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:19.063600 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:19.064395 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:19.064420 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.065673 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.065678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:19.067106 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:19.067118 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.067876 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.067883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:19.068112 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:19.068879 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:47:19.068882 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:19.068927 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:47:19.069010 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:19.069391 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:19.069403 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:19.069411 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:47:19.069604 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 02:47:19.070014 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":293,"threads_started":1}
I20260929 02:47:19.072434 10031 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34735
I20260929 02:47:19.072805 10031 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:19.073676 10031 heartbeater.cc:511] Master 127.0.97.62:34735 requested a full tablet report, sending...
I20260929 02:47:19.075161 9714 ts_manager.cc:194] Registered new tserver with Master: 9264cb511f8e41088d409976467d4d7c (127.0.97.2:37313)
I20260929 02:47:19.075994 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:43825
W20260929 02:47:19.397513 10034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:19.397809 10034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:19.397976 10034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:19.405200 10034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:19.405301 10034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:19.405402 10034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:47:19.414685 10034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:19.415951 10034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:19.417717 10034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:19.430933 10040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:19.431511 10042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:19.431471 10039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:19.431746 10034 server_base.cc:1034] running on GCE node
I20260929 02:47:19.432464 10034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:19.433614 10034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:19.434881 10034 hybrid_clock.cc:648] HybridClock initialized: now 1790650039434836 us; error 66 us; skew 500 ppm
I20260929 02:47:19.435256 10034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:19.438020 10034 webserver.cc:460] Webserver started at http://127.0.97.3:43219/ using document root <none> and password file <none>
I20260929 02:47:19.439044 10034 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:19.439195 10034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:19.439553 10034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:19.442806 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "a6af62fd3d3a40f6b557536067b4b520"
format_stamp: "Formatted at 2026-09-29 02:47:19 on dist-test-slave-6tjh"
server_key: "264f89483426b8abcc3a99f66dd47a2d"
server_key_iv: "c993b38d40af6b11b96ef41ce482d7ea"
server_key_version: "encryptionkey@0"
I20260929 02:47:19.443964 10034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a6af62fd3d3a40f6b557536067b4b520"
format_stamp: "Formatted at 2026-09-29 02:47:19 on dist-test-slave-6tjh"
server_key: "264f89483426b8abcc3a99f66dd47a2d"
server_key_iv: "c993b38d40af6b11b96ef41ce482d7ea"
server_key_version: "encryptionkey@0"
I20260929 02:47:19.449873 10034 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:47:19.453629 10048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:19.455050 10034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:47:19.455252 10034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "a6af62fd3d3a40f6b557536067b4b520"
format_stamp: "Formatted at 2026-09-29 02:47:19 on dist-test-slave-6tjh"
server_key: "264f89483426b8abcc3a99f66dd47a2d"
server_key_iv: "c993b38d40af6b11b96ef41ce482d7ea"
server_key_version: "encryptionkey@0"
I20260929 02:47:19.455492 10034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:19.500710 10034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:19.501600 10034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:19.501889 10034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:19.502877 10034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:19.504339 10034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:19.504463 10034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:19.504613 10034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:19.504699 10034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:19.538354 10034 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39005
I20260929 02:47:19.538407 10160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39005 every 8 connection(s)
I20260929 02:47:19.539659 10034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:47:19.540901 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10034
I20260929 02:47:19.541085 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:47:19.541608 388 external_mini_cluster.cc:1428] Setting key 0c65a3621e0c9281e610b3dc47fe5007
I20260929 02:47:19.552888 9902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:19.542659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38461 (local address 127.0.97.62:34735)
0929 02:47:19.542856 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:19.542863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:19.544464 (+ 1601us) server_negotiation.cc:407] Connection header received
0929 02:47:19.546377 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:19.546382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:19.546467 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:19.546589 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:19.547639 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.548403 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.550044 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.550462 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.550597 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:19.552052 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:19.552058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:19.552063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:19.552090 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:19.552118 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:19.552129 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:19.552238 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:19.552453 (+ 215us) server_negotiation.cc:299] Negotiation successful
0929 02:47:19.552571 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:47:19.553884 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:19.543296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34735 (local address 127.0.97.3:38461)
0929 02:47:19.544303 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:19.544367 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:19.546120 (+ 1753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:19.546827 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:19.546836 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:19.546932 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:19.547483 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:19.547499 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.548584 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.548590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:19.549877 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:19.549910 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.550681 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.550688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:19.550934 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:19.551732 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:47:19.551735 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:19.551781 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:47:19.551862 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:19.552239 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:19.552250 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:19.552260 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:47:19.552445 (+ 185us) client_negotiation.cc:240] Negotiation successful
0929 02:47:19.552875 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":372,"threads_started":1}
I20260929 02:47:19.555356 10161 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34735
I20260929 02:47:19.555732 10161 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:19.556767 10161 heartbeater.cc:511] Master 127.0.97.62:34735 requested a full tablet report, sending...
I20260929 02:47:19.558367 9714 ts_manager.cc:194] Registered new tserver with Master: a6af62fd3d3a40f6b557536067b4b520 (127.0.97.3:39005)
I20260929 02:47:19.559293 9714 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:38461
I20260929 02:47:19.569737 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:47:19.576037 9902 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:19.574397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.0.97.62:34735)
0929 02:47:19.574571 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:19.574576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:19.575085 (+ 509us) server_negotiation.cc:407] Connection header received
0929 02:47:19.575233 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:19.575236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:19.575322 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:19.575416 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:19.575856 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56542: BlockingRecv error: recv got EOF from 127.0.0.1:56542 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:47:19.576620 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9773
I20260929 02:47:19.584780 10170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:19.577655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.0.97.1:44093)
0929 02:47:19.578218 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:19.578240 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:19.578267 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:47:19.578321 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:19.578327 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:19.578622 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:19.578746 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:19.579127 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.580222 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.581540 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.582130 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.582258 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:19.583262 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:19.583269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:19.583283 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:19.583889 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:19.583932 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:19.583950 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:19.584040 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:19.584221 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 02:47:19.584317 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":304,"threads_started":1}
I20260929 02:47:19.586025 9895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:19.667622 9900 heartbeater.cc:503] Master 127.0.97.62:34735 was elected leader, sending a full tablet report...
I20260929 02:47:19.668987 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9773
W20260929 02:47:19.685122 9700 connection.cc:535] server connection from 127.0.97.1:48267 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:47:19.685429 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9904
I20260929 02:47:19.693584 10172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:19.686383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47374 (local address 127.0.97.2:37313)
0929 02:47:19.686855 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:19.686862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:19.686889 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:47:19.686951 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:19.686958 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:19.687178 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:19.687327 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:19.687769 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.688900 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.690180 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.690752 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.690866 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:19.691920 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:19.691927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:19.691941 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:19.692534 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:19.692653 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:19.692686 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:19.692779 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:19.692996 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 02:47:19.693194 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":236,"threads_started":1}
I20260929 02:47:19.694902 10026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:19.779851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9904
I20260929 02:47:19.795567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10034
I20260929 02:47:19.803602 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:19.796528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.0.97.3:39005)
0929 02:47:19.797234 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:19.797240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:19.797270 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:47:19.797331 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:19.797337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:19.797563 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:19.797694 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:19.798063 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.799095 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.800349 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:19.801018 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:19.801111 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:19.802211 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:19.802219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:19.802233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:19.802821 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:19.802866 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:19.802885 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:19.802983 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:19.803095 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:47:19.803223 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":388,"threads_started":1}
I20260929 02:47:19.805027 10156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:19.886945 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10034
I20260929 02:47:19.902622 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 9683
I20260929 02:47:19.903875 9744 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:20.004408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 9683
2026-09-29T02:47:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2494 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T02:47:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:47:20Z Disabled control of system clock
I20260929 02:47:20.077616 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35909
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:47:20.408025 10180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:20.408327 10180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:20.408414 10180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:20.415742 10180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:47:20.415896 10180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:20.415935 10180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:20.415984 10180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:47:20.416031 10180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:47:20.425184 10180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:20.426461 10180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:20.428154 10180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:20.441445 10185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:20.441478 10186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:20.441447 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:20.442634 10180 server_base.cc:1034] running on GCE node
I20260929 02:47:20.443545 10180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:20.445189 10180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:20.446429 10180 hybrid_clock.cc:648] HybridClock initialized: now 1790650040446387 us; error 45 us; skew 500 ppm
I20260929 02:47:20.446790 10180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:20.449188 10180 webserver.cc:460] Webserver started at http://127.0.97.62:43085/ using document root <none> and password file <none>
I20260929 02:47:20.449925 10180 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:20.450057 10180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:20.450364 10180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:20.453302 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "1361c38d78f64eb4890f35608a24961a"
format_stamp: "Formatted at 2026-09-29 02:47:20 on dist-test-slave-6tjh"
server_key: "01d74c70a2b410ffd7eb33921f11d3f9"
server_key_iv: "f3701bdd06ea999287c512e1fd2d30e6"
server_key_version: "encryptionkey@0"
I20260929 02:47:20.454115 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "1361c38d78f64eb4890f35608a24961a"
format_stamp: "Formatted at 2026-09-29 02:47:20 on dist-test-slave-6tjh"
server_key: "01d74c70a2b410ffd7eb33921f11d3f9"
server_key_iv: "f3701bdd06ea999287c512e1fd2d30e6"
server_key_version: "encryptionkey@0"
I20260929 02:47:20.459093 10180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:47:20.462536 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:20.463902 10180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:47:20.464097 10180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "1361c38d78f64eb4890f35608a24961a"
format_stamp: "Formatted at 2026-09-29 02:47:20 on dist-test-slave-6tjh"
server_key: "01d74c70a2b410ffd7eb33921f11d3f9"
server_key_iv: "f3701bdd06ea999287c512e1fd2d30e6"
server_key_version: "encryptionkey@0"
I20260929 02:47:20.464322 10180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:20.470113 10180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:20.470870 10180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:20.471159 10180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:20.489830 10180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35909
I20260929 02:47:20.489921 10245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35909 every 8 connection(s)
I20260929 02:47:20.491264 10180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:47:20.495074 10246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:47:20.500779 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10180
I20260929 02:47:20.500999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:47:20.501806 388 external_mini_cluster.cc:1428] Setting key 2bfd665a889e3ad5fdc119b8353bf9d3
I20260929 02:47:20.506165 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a: Bootstrap starting.
I20260929 02:47:20.509752 10246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a: Neither blocks nor log segments found. Creating new log.
I20260929 02:47:20.510162 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:20.503927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.0.97.62:35909)
0929 02:47:20.504927 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:20.504938 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:20.504978 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:47:20.506606 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:20.506628 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:20.507044 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:20.507364 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:20.508485 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:20.508491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:20.508503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:20.508516 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:20.508627 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:20.508645 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:20.508834 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:20.508930 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 02:47:20.509200 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":352,"threads_started":1}
I20260929 02:47:20.510905 10246 log.cc:824] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:47:20.513675 10246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a: No bootstrap required, opened a new log
I20260929 02:47:20.520678 10246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1361c38d78f64eb4890f35608a24961a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35909 } }
I20260929 02:47:20.521014 10246 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:47:20.521104 10246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1361c38d78f64eb4890f35608a24961a, State: Initialized, Role: FOLLOWER
I20260929 02:47:20.521656 10246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1361c38d78f64eb4890f35608a24961a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35909 } }
I20260929 02:47:20.521888 10246 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:47:20.522009 10246 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:47:20.522219 10246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:47:20.524307 10246 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1361c38d78f64eb4890f35608a24961a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35909 } }
I20260929 02:47:20.524808 10246 leader_election.cc:302] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1361c38d78f64eb4890f35608a24961a; no voters:
I20260929 02:47:20.525456 10246 leader_election.cc:288] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:47:20.525594 10251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:47:20.526424 10251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [term 1 LEADER]: Becoming Leader. State: Replica: 1361c38d78f64eb4890f35608a24961a, State: Running, Role: LEADER
I20260929 02:47:20.526949 10251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1361c38d78f64eb4890f35608a24961a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35909 } }
I20260929 02:47:20.527570 10246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:47:20.529852 10253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1361c38d78f64eb4890f35608a24961a. Latest consensus state: current_term: 1 leader_uuid: "1361c38d78f64eb4890f35608a24961a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1361c38d78f64eb4890f35608a24961a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35909 } } }
I20260929 02:47:20.530288 10253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:20.531912 10259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:47:20.531872 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1361c38d78f64eb4890f35608a24961a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1361c38d78f64eb4890f35608a24961a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35909 } } }
I20260929 02:47:20.532239 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:20.537317 10259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:47:20.538040 10268 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:47:20.546502 10259 catalog_manager.cc:1347] Generated new cluster ID: 91c0ccc5d2c048feb04a72d4dab07d24
I20260929 02:47:20.546624 10259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:47:20.555414 10259 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:47:20.555873 10259 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:47:20.562379 10259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1361c38d78f64eb4890f35608a24961a: Generated new TSK 0
I20260929 02:47:20.563417 10259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:47:20.570384 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35909
--builtin_ntp_servers=127.0.97.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 02:47:20.901533 10270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:20.901808 10270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:20.901974 10270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:20.909241 10270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:20.909343 10270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:20.909515 10270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:47:20.918695 10270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:20.920027 10270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:20.921846 10270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:20.934991 10275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:20.934948 10278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:20.935029 10276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:20.935783 10270 server_base.cc:1034] running on GCE node
I20260929 02:47:20.936439 10270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:20.937412 10270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:20.938645 10270 hybrid_clock.cc:648] HybridClock initialized: now 1790650040938586 us; error 57 us; skew 500 ppm
I20260929 02:47:20.938974 10270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:20.941300 10270 webserver.cc:460] Webserver started at http://127.0.97.1:36663/ using document root <none> and password file <none>
I20260929 02:47:20.941967 10270 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:20.942095 10270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:20.942410 10270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:20.946760 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "d59609732e4a460a92e61cb17bc76d1d"
format_stamp: "Formatted at 2026-09-29 02:47:20 on dist-test-slave-6tjh"
server_key: "6063516de1cb3b0d3635a0fb6ad3c096"
server_key_iv: "36641b2c23c799f2a7ff1c57df87c5c7"
server_key_version: "encryptionkey@0"
I20260929 02:47:20.947525 10270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d59609732e4a460a92e61cb17bc76d1d"
format_stamp: "Formatted at 2026-09-29 02:47:20 on dist-test-slave-6tjh"
server_key: "6063516de1cb3b0d3635a0fb6ad3c096"
server_key_iv: "36641b2c23c799f2a7ff1c57df87c5c7"
server_key_version: "encryptionkey@0"
I20260929 02:47:20.952651 10270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:47:20.956509 10284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:20.957836 10270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:47:20.958009 10270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "d59609732e4a460a92e61cb17bc76d1d"
format_stamp: "Formatted at 2026-09-29 02:47:20 on dist-test-slave-6tjh"
server_key: "6063516de1cb3b0d3635a0fb6ad3c096"
server_key_iv: "36641b2c23c799f2a7ff1c57df87c5c7"
server_key_version: "encryptionkey@0"
I20260929 02:47:20.958235 10270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:20.963550 10270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:20.964344 10270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:20.964680 10270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:20.965618 10270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:20.966989 10270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:20.967108 10270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:20.967249 10270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:20.967350 10270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:21.001896 10270 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41383
I20260929 02:47:21.001935 10396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41383 every 8 connection(s)
I20260929 02:47:21.003255 10270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:47:21.006026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10270
I20260929 02:47:21.006251 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:47:21.006944 388 external_mini_cluster.cc:1428] Setting key 4a497b47cbe111271c1f8ad140f9eabc
I20260929 02:47:21.010124 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35909
--builtin_ntp_servers=127.0.97.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:47:21.012480 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:21.005944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38597 (local address 127.0.97.62:35909)
0929 02:47:21.006174 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:21.006182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:21.007830 (+ 1648us) server_negotiation.cc:407] Connection header received
0929 02:47:21.010023 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:21.010027 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:21.010075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:21.010159 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:21.011764 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:21.011769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:21.011772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:21.011790 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:21.011821 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:21.011832 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:21.011933 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:21.012167 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 02:47:21.012234 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:47:21.013772 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:21.006553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35909 (local address 127.0.97.1:38597)
0929 02:47:21.007666 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:21.007703 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:21.009706 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:21.010448 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:21.010462 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:21.010595 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:21.011399 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:47:21.011405 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:21.011459 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:47:21.011584 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:21.011930 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:21.011942 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:21.011954 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 02:47:21.012235 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 02:47:21.012496 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":306,"threads_started":1}
I20260929 02:47:21.015275 10397 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35909
I20260929 02:47:21.015583 10397 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:21.016189 10397 heartbeater.cc:511] Master 127.0.97.62:35909 requested a full tablet report, sending...
I20260929 02:47:21.018215 10211 ts_manager.cc:194] Registered new tserver with Master: d59609732e4a460a92e61cb17bc76d1d (127.0.97.1:41383)
W20260929 02:47:21.342075 10400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:21.342363 10400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:21.342509 10400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:21.349895 10400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:21.350004 10400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:21.350171 10400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:47:21.360132 10400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:21.361680 10400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:21.363673 10400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:21.378005 10406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:21.378105 10408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:21.378336 10405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:21.379446 10400 server_base.cc:1034] running on GCE node
I20260929 02:47:21.380048 10400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:21.381072 10400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:21.382311 10400 hybrid_clock.cc:648] HybridClock initialized: now 1790650041382258 us; error 40 us; skew 500 ppm
I20260929 02:47:21.382624 10400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:21.385082 10400 webserver.cc:460] Webserver started at http://127.0.97.2:34515/ using document root <none> and password file <none>
I20260929 02:47:21.385762 10400 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:21.385912 10400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:21.386236 10400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:21.389120 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "6113c284456541819a292673d4f1ab55"
format_stamp: "Formatted at 2026-09-29 02:47:21 on dist-test-slave-6tjh"
server_key: "9630389a555648bb01b74d8af433a36d"
server_key_iv: "b89252406c3afd70e6fdea2e7b7a7433"
server_key_version: "encryptionkey@0"
I20260929 02:47:21.389902 10400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6113c284456541819a292673d4f1ab55"
format_stamp: "Formatted at 2026-09-29 02:47:21 on dist-test-slave-6tjh"
server_key: "9630389a555648bb01b74d8af433a36d"
server_key_iv: "b89252406c3afd70e6fdea2e7b7a7433"
server_key_version: "encryptionkey@0"
I20260929 02:47:21.394930 10400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:47:21.398353 10414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:21.399676 10400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:47:21.399863 10400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "6113c284456541819a292673d4f1ab55"
format_stamp: "Formatted at 2026-09-29 02:47:21 on dist-test-slave-6tjh"
server_key: "9630389a555648bb01b74d8af433a36d"
server_key_iv: "b89252406c3afd70e6fdea2e7b7a7433"
server_key_version: "encryptionkey@0"
I20260929 02:47:21.400103 10400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:21.405591 10400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:21.406416 10400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:21.406693 10400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:21.407614 10400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:21.409058 10400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:21.409179 10400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:21.409323 10400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:21.409413 10400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:21.442592 10400 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39639
I20260929 02:47:21.442678 10526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39639 every 8 connection(s)
I20260929 02:47:21.443934 10400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:47:21.444521 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10400
I20260929 02:47:21.444877 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:47:21.445426 388 external_mini_cluster.cc:1428] Setting key bc1a12b07f7c62912b9d67a0de198947
I20260929 02:47:21.448593 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35909
--builtin_ntp_servers=127.0.97.20:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 02:47:21.454172 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:21.446912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57679 (local address 127.0.97.62:35909)
0929 02:47:21.447111 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:21.447117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:21.449015 (+ 1898us) server_negotiation.cc:407] Connection header received
0929 02:47:21.451820 (+ 2805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:21.451824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:21.451878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:21.451970 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:21.453519 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:21.453523 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:21.453526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:21.453542 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:21.453573 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:21.453585 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:21.453694 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:21.453907 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 02:47:21.453957 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:47:21.455552 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:21.447633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35909 (local address 127.0.97.2:57679)
0929 02:47:21.448858 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:21.448908 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:21.451500 (+ 2592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:21.452242 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:21.452256 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:21.452399 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:21.453144 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:47:21.453150 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:21.453203 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:47:21.453331 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:21.453684 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:21.453696 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:21.453706 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:47:21.453985 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 02:47:21.454265 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":304,"threads_started":1}
I20260929 02:47:21.457384 10527 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35909
I20260929 02:47:21.457753 10527 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:21.458371 10527 heartbeater.cc:511] Master 127.0.97.62:35909 requested a full tablet report, sending...
I20260929 02:47:21.459915 10211 ts_manager.cc:194] Registered new tserver with Master: 6113c284456541819a292673d4f1ab55 (127.0.97.2:39639)
W20260929 02:47:21.793397 10530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:21.793704 10530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:21.793874 10530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:21.801273 10530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:21.801391 10530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:21.801611 10530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:47:21.811221 10530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35909
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:21.812873 10530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:21.814771 10530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:21.828017 10535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:21.828056 10538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:21.827993 10536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:21.828969 10530 server_base.cc:1034] running on GCE node
I20260929 02:47:21.829687 10530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:21.830651 10530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:21.831892 10530 hybrid_clock.cc:648] HybridClock initialized: now 1790650041831855 us; error 33 us; skew 500 ppm
I20260929 02:47:21.832291 10530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:47:21.834735 10530 webserver.cc:460] Webserver started at http://127.0.97.3:35793/ using document root <none> and password file <none>
I20260929 02:47:21.835422 10530 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:21.835554 10530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:21.835873 10530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:21.838829 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "4e9cd77325cf4523acd5d757b43d8bb7"
format_stamp: "Formatted at 2026-09-29 02:47:21 on dist-test-slave-6tjh"
server_key: "69bb70ff2deb4b619d7c728ee8b3b6cd"
server_key_iv: "70138099d989b24eda7090d50ba6e622"
server_key_version: "encryptionkey@0"
I20260929 02:47:21.839581 10530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4e9cd77325cf4523acd5d757b43d8bb7"
format_stamp: "Formatted at 2026-09-29 02:47:21 on dist-test-slave-6tjh"
server_key: "69bb70ff2deb4b619d7c728ee8b3b6cd"
server_key_iv: "70138099d989b24eda7090d50ba6e622"
server_key_version: "encryptionkey@0"
I20260929 02:47:21.844818 10530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 02:47:21.848363 10544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:21.849828 10530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:47:21.849994 10530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "4e9cd77325cf4523acd5d757b43d8bb7"
format_stamp: "Formatted at 2026-09-29 02:47:21 on dist-test-slave-6tjh"
server_key: "69bb70ff2deb4b619d7c728ee8b3b6cd"
server_key_iv: "70138099d989b24eda7090d50ba6e622"
server_key_version: "encryptionkey@0"
I20260929 02:47:21.850229 10530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:21.855818 10530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:21.856669 10530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:21.856930 10530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:21.857833 10530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:21.859165 10530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:21.859292 10530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:21.859421 10530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:21.859513 10530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:21.892726 10530 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:32995
I20260929 02:47:21.892789 10656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:32995 every 8 connection(s)
I20260929 02:47:21.894074 10530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:47:21.895480 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10530
I20260929 02:47:21.895694 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:47:21.896198 388 external_mini_cluster.cc:1428] Setting key 43915ad507c1614bb75658a4c2999ce7
I20260929 02:47:21.902663 10249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:21.896748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42561 (local address 127.0.97.62:35909)
0929 02:47:21.896993 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:21.897000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:21.898756 (+ 1756us) server_negotiation.cc:407] Connection header received
0929 02:47:21.900455 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:21.900460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:21.900525 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:21.900640 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:21.901954 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:21.901960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:21.901965 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:21.901982 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:21.902021 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:21.902033 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:21.902152 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:21.902342 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 02:47:21.902407 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:47:21.903628 10659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:21.897469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35909 (local address 127.0.97.3:42561)
0929 02:47:21.898608 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:21.898660 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:21.900185 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:21.900865 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:21.900874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:21.900968 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:21.901669 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:47:21.901671 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:21.901709 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:47:21.901794 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:21.902124 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:21.902135 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:21.902142 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:47:21.902423 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 02:47:21.902674 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":393,"threads_started":1}
I20260929 02:47:21.905071 10657 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35909
I20260929 02:47:21.905336 10657 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:21.905872 10657 heartbeater.cc:511] Master 127.0.97.62:35909 requested a full tablet report, sending...
I20260929 02:47:21.907177 10211 ts_manager.cc:194] Registered new tserver with Master: 4e9cd77325cf4523acd5d757b43d8bb7 (127.0.97.3:32995)
I20260929 02:47:21.911446 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 02:47:21.917505 10249 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:47:21.915746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34234 (local address 127.0.97.62:35909)
0929 02:47:21.915908 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:21.915913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:21.916608 (+ 695us) server_negotiation.cc:407] Connection header received
0929 02:47:21.916788 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:21.916791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:21.916838 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:21.916917 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:21.917221 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34234: BlockingRecv error: recv got EOF from 127.0.0.1:34234 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 02:47:21.919286 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10270
I20260929 02:47:21.922652 10666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:21.920261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.0.97.1:41383)
0929 02:47:21.920764 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:21.920771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:21.920797 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:47:21.920852 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:21.920858 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:21.921057 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:21.921180 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:21.922042 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:21.922050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:21.922062 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:21.922078 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:21.922130 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:21.922149 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:21.922280 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:21.922332 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 02:47:21.922429 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
I20260929 02:47:21.923840 10392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:22.005817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10270
I20260929 02:47:22.018872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10400
I20260929 02:47:22.022571 10668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:22.019776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.0.97.2:39639)
0929 02:47:22.020280 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:22.020287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:22.020313 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:47:22.020372 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:22.020378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:22.020656 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:22.020827 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:22.021880 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:22.021889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:22.021904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:22.021925 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:22.021980 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:22.021999 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:22.022140 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:22.022236 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 02:47:22.022330 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":242,"threads_started":1}
I20260929 02:47:22.023839 10522 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:22.105544 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10400
I20260929 02:47:22.121028 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10530
I20260929 02:47:22.124425 10670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:22.121944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.0.97.3:32995)
0929 02:47:22.122554 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:22.122561 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:22.122591 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:47:22.122653 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:22.122658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:22.122880 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:22.123007 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:22.123835 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:22.123842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:22.123853 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:47:22.123864 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:22.123903 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:47:22.123916 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:22.124052 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:22.124095 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 02:47:22.124172 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":314,"threads_started":1}
I20260929 02:47:22.125692 10652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:22.207151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10530
I20260929 02:47:22.223218 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10180
I20260929 02:47:22.224449 10241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:22.325119 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10180
2026-09-29T02:47:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2320 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:47:22 dist-test-slave-6tjh krb5kdc[10675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:47:22 dist-test-slave-6tjh krb5kdc[10675](info): set up 2 sockets
Sep 29 02:47:22 dist-test-slave-6tjh krb5kdc[10675](info): commencing operation
krb5kdc: starting...
W20260929 02:47:23.426927 388 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:47:23 dist-test-slave-6tjh krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:47:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:47:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:47:23.672449 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38617
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38617
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:47:23.994787 10688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:23.995074 10688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:23.995244 10688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:24.002424 10688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:47:24.002529 10688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:24.002621 10688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:24.002723 10688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:47:24.002807 10688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:47:24.012593 10688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:24.013922 10688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:24.015213 10688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:24.026708 10693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:24.026677 10694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:24.026628 10696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:24.027458 10688 server_base.cc:1034] running on GCE node
I20260929 02:47:24.028219 10688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:24.029641 10688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:24.030834 10688 hybrid_clock.cc:648] HybridClock initialized: now 1790650044030791 us; error 35 us; skew 500 ppm
I20260929 02:47:24.031126 10688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:24 dist-test-slave-6tjh krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:24.037998 10688 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:47:24.039547 10688 webserver.cc:460] Webserver started at http://127.0.97.62:34767/ using document root <none> and password file <none>
I20260929 02:47:24.040252 10688 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:24.040385 10688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:24.040762 10688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:24.043664 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "46ea770019c14409b66e8fb5f4bc2fb3"
format_stamp: "Formatted at 2026-09-29 02:47:24 on dist-test-slave-6tjh"
server_key: "12da39aa02cb2b2eae9d3d7ca827d506"
server_key_iv: "8f4e9c81764b5dc5fa633d3b7efe0eef"
server_key_version: "encryptionkey@0"
I20260929 02:47:24.044407 10688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "46ea770019c14409b66e8fb5f4bc2fb3"
format_stamp: "Formatted at 2026-09-29 02:47:24 on dist-test-slave-6tjh"
server_key: "12da39aa02cb2b2eae9d3d7ca827d506"
server_key_iv: "8f4e9c81764b5dc5fa633d3b7efe0eef"
server_key_version: "encryptionkey@0"
I20260929 02:47:24.049508 10688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 02:47:24.053136 10703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:24.054448 10688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:47:24.054632 10688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "46ea770019c14409b66e8fb5f4bc2fb3"
format_stamp: "Formatted at 2026-09-29 02:47:24 on dist-test-slave-6tjh"
server_key: "12da39aa02cb2b2eae9d3d7ca827d506"
server_key_iv: "8f4e9c81764b5dc5fa633d3b7efe0eef"
server_key_version: "encryptionkey@0"
I20260929 02:47:24.054827 10688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:24.071589 10688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:24.075657 10688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:24.075951 10688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:24.094116 10688 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38617
I20260929 02:47:24.094138 10754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38617 every 8 connection(s)
I20260929 02:47:24.095523 10688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:47:24.099408 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:47:24.104214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10688
I20260929 02:47:24.104482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:47:24.105139 388 external_mini_cluster.cc:1428] Setting key 38f0138028e1010484b71756820dff2c
I20260929 02:47:24.110325 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3: Bootstrap starting.
I20260929 02:47:24.114188 10755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3: Neither blocks nor log segments found. Creating new log.
I20260929 02:47:24.115414 10755 log.cc:824] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:47:24.118407 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3: No bootstrap required, opened a new log
Sep 29 02:47:24 dist-test-slave-6tjh krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:47:24.126322 10755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38617 } }
I20260929 02:47:24.126713 10755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:47:24.126816 10755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ea770019c14409b66e8fb5f4bc2fb3, State: Initialized, Role: FOLLOWER
I20260929 02:47:24.127380 10755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38617 } }
I20260929 02:47:24.127602 10755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:47:24.127733 10755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:47:24.127924 10755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:47:24.128059 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:24.107209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.0.97.62:38617)
0929 02:47:24.108101 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:24.108113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:24.108161 (+ 48us) server_negotiation.cc:407] Connection header received
0929 02:47:24.109882 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:24.109911 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:24.110328 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:24.110707 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:24.112531 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:24.113852 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:24.115644 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:24.116324 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:24.116613 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:24.120441 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:24.120451 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:24.120465 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:24.120507 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:24.123766 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:24.124451 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:24.124457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:24.124463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:24.124679 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:24.125063 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:24.125068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:24.125072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:24.125599 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:24.125881 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:24.126280 (+ 399us) server_negotiation.cc:299] Negotiation successful
0929 02:47:24.126744 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":427,"threads_started":1}
I20260929 02:47:24.130307 10755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38617 } }
I20260929 02:47:24.130779 10755 leader_election.cc:302] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46ea770019c14409b66e8fb5f4bc2fb3; no voters:
I20260929 02:47:24.131318 10755 leader_election.cc:288] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:47:24.131484 10760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:47:24.132510 10760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [term 1 LEADER]: Becoming Leader. State: Replica: 46ea770019c14409b66e8fb5f4bc2fb3, State: Running, Role: LEADER
I20260929 02:47:24.133138 10760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38617 } }
I20260929 02:47:24.133440 10755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:47:24.135885 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46ea770019c14409b66e8fb5f4bc2fb3. Latest consensus state: current_term: 1 leader_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38617 } } }
I20260929 02:47:24.136286 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:24.137504 10761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46ea770019c14409b66e8fb5f4bc2fb3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38617 } } }
I20260929 02:47:24.137763 10761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:47:24.138334 10768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:47:24.143635 10768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:47:24.152119 10768 catalog_manager.cc:1347] Generated new cluster ID: c0879d973eba429f804716db0ee52431
I20260929 02:47:24.152278 10768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:47:24.179584 10768 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:47:24.180946 10768 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:47:24.188098 10768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3: Generated new TSK 0
I20260929 02:47:24.188997 10768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:47:24.255971 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38617
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:47:24.592463 10781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:24.592983 10781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:24.593166 10781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:24.601280 10781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:24.601441 10781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:24.601624 10781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:47:24.611614 10781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:24.613032 10781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:24.614351 10781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:24.627843 10787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:24.627831 10786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:24.628515 10789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:24.628732 10781 server_base.cc:1034] running on GCE node
I20260929 02:47:24.629490 10781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:24.630479 10781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:24.631713 10781 hybrid_clock.cc:648] HybridClock initialized: now 1790650044631654 us; error 55 us; skew 500 ppm
I20260929 02:47:24.632052 10781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:24 dist-test-slave-6tjh krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:24.635404 10781 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:47:24.636888 10781 webserver.cc:460] Webserver started at http://127.0.97.1:34357/ using document root <none> and password file <none>
I20260929 02:47:24.637599 10781 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:24.637748 10781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:24.638064 10781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:24.641026 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "80c575191b4b4c00869247e9a88a9fc9"
format_stamp: "Formatted at 2026-09-29 02:47:24 on dist-test-slave-6tjh"
server_key: "23c5b41c641cb2f44c6df4321ae1f6c3"
server_key_iv: "68e4ee951325a01220455669cae0b8cf"
server_key_version: "encryptionkey@0"
I20260929 02:47:24.641775 10781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "80c575191b4b4c00869247e9a88a9fc9"
format_stamp: "Formatted at 2026-09-29 02:47:24 on dist-test-slave-6tjh"
server_key: "23c5b41c641cb2f44c6df4321ae1f6c3"
server_key_iv: "68e4ee951325a01220455669cae0b8cf"
server_key_version: "encryptionkey@0"
I20260929 02:47:24.646929 10781 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:47:24.650774 10796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:24.652199 10781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:47:24.652405 10781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "80c575191b4b4c00869247e9a88a9fc9"
format_stamp: "Formatted at 2026-09-29 02:47:24 on dist-test-slave-6tjh"
server_key: "23c5b41c641cb2f44c6df4321ae1f6c3"
server_key_iv: "68e4ee951325a01220455669cae0b8cf"
server_key_version: "encryptionkey@0"
I20260929 02:47:24.652683 10781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:24.678722 10781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:24.682713 10781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:24.683012 10781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:24.683962 10781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:24.685323 10781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:24.685426 10781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:24.685576 10781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:24.685652 10781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:24.719337 10781 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:34203
I20260929 02:47:24.719380 10908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:34203 every 8 connection(s)
I20260929 02:47:24.720726 10781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:47:24.721047 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10781
I20260929 02:47:24.721256 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:47:24.722074 388 external_mini_cluster.cc:1428] Setting key 09ef9e364e3698de6647de1830cbdce9
Sep 29 02:47:24 dist-test-slave-6tjh krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:47:24.752621 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:24.726973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47201 (local address 127.0.97.62:38617)
0929 02:47:24.730657 (+ 3684us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:24.730666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:24.730688 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:47:24.730766 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:24.730772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:24.730859 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:24.731007 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:24.734139 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:24.735560 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:24.738399 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:24.739122 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:24.739275 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:24.744609 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:24.744616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:24.744620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:24.744659 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:24.748486 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:24.749670 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:24.749677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:24.749682 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:24.749825 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:24.750435 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:24.750442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:24.750449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:24.750732 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:24.750888 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:24.751998 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0929 02:47:24.752277 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3523,"thread_start_us":3422,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:47:24.754259 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:24.724452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38617 (local address 127.0.97.1:47201)
0929 02:47:24.725623 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:24.725680 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:24.728368 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:24.731514 (+ 3146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:24.731532 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:24.732716 (+ 1184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:24.733880 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:24.733912 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:24.735744 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:24.735750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:24.738215 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:24.738234 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:24.739375 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:24.739385 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:24.739707 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:24.740691 (+ 984us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:24.740724 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:24.744288 (+ 3564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:24.748727 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:47:24.748738 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:47:24.748759 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:47:24.749473 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:47:24.749958 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:47:24.749966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:47:24.749970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:47:24.750265 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:47:24.750895 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:24.750906 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:24.751455 (+ 549us) client_negotiation.cc:769] Sending connection context
0929 02:47:24.751838 (+ 383us) client_negotiation.cc:240] Negotiation successful
0929 02:47:24.752674 (+ 836us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":390,"threads_started":1}
I20260929 02:47:24.755961 10909 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38617
I20260929 02:47:24.756335 10909 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:24.757196 10909 heartbeater.cc:511] Master 127.0.97.62:38617 requested a full tablet report, sending...
I20260929 02:47:24.759541 10720 ts_manager.cc:194] Registered new tserver with Master: 80c575191b4b4c00869247e9a88a9fc9 (127.0.97.1:34203)
I20260929 02:47:24.761365 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:47:24.783818 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38617
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:47:25.119648 10915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:25.120009 10915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:25.120201 10915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:25.127812 10915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:25.127980 10915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:25.128175 10915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:47:25.138501 10915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:25.139830 10915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:25.141372 10915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:25.155138 10923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:25.155207 10920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:25.155174 10921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:25.155992 10915 server_base.cc:1034] running on GCE node
I20260929 02:47:25.156798 10915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:25.157851 10915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:25.159121 10915 hybrid_clock.cc:648] HybridClock initialized: now 1790650045159055 us; error 75 us; skew 500 ppm
I20260929 02:47:25.159508 10915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:25 dist-test-slave-6tjh krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:25.162576 10915 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:47:25.164052 10915 webserver.cc:460] Webserver started at http://127.0.97.2:44021/ using document root <none> and password file <none>
I20260929 02:47:25.164814 10915 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:25.164958 10915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:25.165247 10915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:25.168110 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "9fc499bd97494cd1b6582c47677ae744"
format_stamp: "Formatted at 2026-09-29 02:47:25 on dist-test-slave-6tjh"
server_key: "3f13009f8d11b46a654855d4245089cf"
server_key_iv: "915fdbbd6db9cef2c885048f44a8cb44"
server_key_version: "encryptionkey@0"
I20260929 02:47:25.168995 10915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9fc499bd97494cd1b6582c47677ae744"
format_stamp: "Formatted at 2026-09-29 02:47:25 on dist-test-slave-6tjh"
server_key: "3f13009f8d11b46a654855d4245089cf"
server_key_iv: "915fdbbd6db9cef2c885048f44a8cb44"
server_key_version: "encryptionkey@0"
I20260929 02:47:25.174067 10915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 02:47:25.177379 10930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:25.178695 10915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:47:25.178875 10915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "9fc499bd97494cd1b6582c47677ae744"
format_stamp: "Formatted at 2026-09-29 02:47:25 on dist-test-slave-6tjh"
server_key: "3f13009f8d11b46a654855d4245089cf"
server_key_iv: "915fdbbd6db9cef2c885048f44a8cb44"
server_key_version: "encryptionkey@0"
I20260929 02:47:25.179064 10915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:25.206152 10915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:25.210215 10915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:25.210526 10915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:25.211642 10915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:25.213038 10915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:25.213146 10915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:25.213299 10915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:25.213372 10915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:25.250725 10915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39233
I20260929 02:47:25.250770 11042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39233 every 8 connection(s)
I20260929 02:47:25.252056 10915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:47:25.256332 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 10915
I20260929 02:47:25.256587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:47:25.257169 388 external_mini_cluster.cc:1428] Setting key 15392ab5a73b9e404f627ffe0e7aa3e5
Sep 29 02:47:25 dist-test-slave-6tjh krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:47:25.275914 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.254753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59581 (local address 127.0.97.62:38617)
0929 02:47:25.255205 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:25.255212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:25.256133 (+ 921us) server_negotiation.cc:407] Connection header received
0929 02:47:25.258567 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:25.258572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:25.258641 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:25.258754 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:25.260883 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.261593 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.264009 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.264439 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.264533 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.268780 (+ 4247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:25.268789 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:25.268798 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:25.268844 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:25.272370 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:25.273284 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:25.273293 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:25.273300 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:25.273456 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:25.273932 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:25.273940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:25.273946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:25.274328 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:25.274473 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:25.275176 (+ 703us) server_negotiation.cc:299] Negotiation successful
0929 02:47:25.275426 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":219,"threads_started":1}
I20260929 02:47:25.276640 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.255174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38617 (local address 127.0.97.2:59581)
0929 02:47:25.255923 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:25.255987 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:25.258269 (+ 2282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:25.259047 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:25.259056 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:25.259815 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:25.260688 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.260717 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.261744 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.261752 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.263842 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.263864 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.264690 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.264699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.265046 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.265838 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:25.265873 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:25.268498 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:25.272600 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:47:25.272618 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:47:25.272633 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:47:25.273129 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:47:25.273592 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:47:25.273596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:47:25.273599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:47:25.273798 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:47:25.274449 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:25.274455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:25.274761 (+ 306us) client_negotiation.cc:769] Sending connection context
0929 02:47:25.275029 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 02:47:25.275426 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":246,"threads_started":1}
I20260929 02:47:25.278177 11043 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38617
I20260929 02:47:25.278492 11043 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:25.279250 11043 heartbeater.cc:511] Master 127.0.97.62:38617 requested a full tablet report, sending...
I20260929 02:47:25.280784 10720 ts_manager.cc:194] Registered new tserver with Master: 9fc499bd97494cd1b6582c47677ae744 (127.0.97.2:39233)
I20260929 02:47:25.281978 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59581
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:47:25.312280 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38617
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:47:25.634779 11049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:47:25.635073 11049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:47:25.635277 11049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:47:25.642745 11049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:47:25.642860 11049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:47:25.643059 11049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:47:25.652853 11049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38617
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:47:25.654161 11049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:47:25.655480 11049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:47:25.668509 11055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:25.668481 11054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:47:25.668711 11057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:47:25.669384 11049 server_base.cc:1034] running on GCE node
I20260929 02:47:25.669931 11049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:47:25.670835 11049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:47:25.672045 11049 hybrid_clock.cc:648] HybridClock initialized: now 1790650045672004 us; error 31 us; skew 500 ppm
I20260929 02:47:25.672351 11049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:47:25 dist-test-slave-6tjh krb5kdc[10675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:47:25.675536 11049 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:47:25.677004 11049 webserver.cc:460] Webserver started at http://127.0.97.3:42595/ using document root <none> and password file <none>
I20260929 02:47:25.677649 11049 fs_manager.cc:359] Metadata directory not provided
I20260929 02:47:25.677744 11049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:47:25.677974 11049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:47:25.680842 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "99530c59667e4bae9a3f9f353c0e018f"
format_stamp: "Formatted at 2026-09-29 02:47:25 on dist-test-slave-6tjh"
server_key: "8036810e421efa846bcfcda7711d259e"
server_key_iv: "c560e36b90b52f43482bf7da0403f359"
server_key_version: "encryptionkey@0"
I20260929 02:47:25.681527 11049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "99530c59667e4bae9a3f9f353c0e018f"
format_stamp: "Formatted at 2026-09-29 02:47:25 on dist-test-slave-6tjh"
server_key: "8036810e421efa846bcfcda7711d259e"
server_key_iv: "c560e36b90b52f43482bf7da0403f359"
server_key_version: "encryptionkey@0"
I20260929 02:47:25.686961 11049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:47:25.690237 11064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:25.691519 11049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:47:25.691638 11049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "99530c59667e4bae9a3f9f353c0e018f"
format_stamp: "Formatted at 2026-09-29 02:47:25 on dist-test-slave-6tjh"
server_key: "8036810e421efa846bcfcda7711d259e"
server_key_iv: "c560e36b90b52f43482bf7da0403f359"
server_key_version: "encryptionkey@0"
I20260929 02:47:25.691785 11049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:47:25.715531 11049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:47:25.719936 11049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:47:25.720235 11049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:47:25.721240 11049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:47:25.722544 11049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:47:25.722612 11049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:25.722738 11049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:47:25.722786 11049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:47:25.758097 11049 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42943
I20260929 02:47:25.758134 11176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42943 every 8 connection(s)
I20260929 02:47:25.759488 11049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:47:25.764416 10909 heartbeater.cc:503] Master 127.0.97.62:38617 was elected leader, sending a full tablet report...
I20260929 02:47:25.764991 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 11049
I20260929 02:47:25.765198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:47:25.765692 388 external_mini_cluster.cc:1428] Setting key aa1cab246834d0ae41e5e78d5b370fb4
Sep 29 02:47:25 dist-test-slave-6tjh krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:47:25.780382 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.762017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37683 (local address 127.0.97.62:38617)
0929 02:47:25.762184 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:25.762188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:25.763260 (+ 1072us) server_negotiation.cc:407] Connection header received
0929 02:47:25.766760 (+ 3500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:25.766765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:25.766855 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:25.766982 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:25.768985 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.769698 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.771194 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.771565 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.771652 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.774858 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:25.774863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:25.774866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:25.774894 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:25.777388 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:25.778049 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:25.778053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:25.778055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:25.778159 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:25.778503 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:25.778506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:25.778508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:25.778756 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:25.778917 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:25.779808 (+ 891us) server_negotiation.cc:299] Negotiation successful
0929 02:47:25.780073 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:47:25.781528 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.762446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38617 (local address 127.0.97.3:37683)
0929 02:47:25.763098 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:25.763142 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:25.766503 (+ 3361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:25.767187 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:25.767195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:25.768067 (+ 872us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:47:25.768844 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.768864 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.769846 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.769850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.771081 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.771093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.771732 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.771736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.771942 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.772661 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:47:25.772682 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:47:25.774683 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:47:25.777534 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:47:25.777541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:47:25.777553 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:47:25.777948 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:47:25.778246 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:47:25.778249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:47:25.778251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:47:25.778416 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:47:25.778899 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:47:25.778906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:47:25.779364 (+ 458us) client_negotiation.cc:769] Sending connection context
0929 02:47:25.779691 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 02:47:25.780205 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":249,"threads_started":1}
I20260929 02:47:25.782977 11177 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38617
I20260929 02:47:25.783274 11177 heartbeater.cc:464] Registering TS with master...
I20260929 02:47:25.783999 11177 heartbeater.cc:511] Master 127.0.97.62:38617 requested a full tablet report, sending...
I20260929 02:47:25.785394 10720 ts_manager.cc:194] Registered new tserver with Master: 99530c59667e4bae9a3f9f353c0e018f (127.0.97.3:42943)
I20260929 02:47:25.786139 10720 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37683
I20260929 02:47:25.793399 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:47:25.810333 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.798179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52698 (local address 127.0.97.62:38617)
0929 02:47:25.798329 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:25.798334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:25.798798 (+ 464us) server_negotiation.cc:407] Connection header received
0929 02:47:25.798942 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:25.798946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:25.799004 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:25.799086 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:25.800424 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.801228 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.802585 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.802994 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.803097 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.805587 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:25.805592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:25.805595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:25.805628 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:25.807993 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:25.808669 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:25.808673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:25.808675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:25.808795 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:25.809148 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:25.809151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:25.809153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:25.809462 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:25.809653 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:25.809938 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 02:47:25.810097 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:47:25.814965 10720 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:47:25.821787 10720 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:47:25.864289 11190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.834071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34203 (local address 127.0.0.1:36576)
0929 02:47:25.854234 (+ 20163us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:25.854266 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:25.854435 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:25.855514 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:25.855525 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:25.855562 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:47:25.855991 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.856009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.858057 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.858062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.860694 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.860706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.862043 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.862050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.863237 (+ 1187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.863296 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 02:47:25.863483 (+ 187us) client_negotiation.cc:240] Negotiation successful
0929 02:47:25.863719 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20020,"mutex_wait_us":150,"thread_start_us":19896,"threads_started":1}
I20260929 02:47:25.864382 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.836513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.0.97.2:39233)
0929 02:47:25.853184 (+ 16671us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:25.853198 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:25.854802 (+ 1604us) server_negotiation.cc:407] Connection header received
0929 02:47:25.854892 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:25.854898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:25.855115 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:25.855271 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:47:25.856578 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.858015 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.861322 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.862990 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.863131 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.863326 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:25.863897 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 02:47:25.864065 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16447,"thread_start_us":16472,"threads_started":1}
I20260929 02:47:25.864993 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.834088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60938 (local address 127.0.97.3:42943)
0929 02:47:25.855942 (+ 21854us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:25.855954 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:25.855988 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:47:25.856109 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:25.856118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:25.856361 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:25.857189 (+ 828us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:47:25.857457 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.858999 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.861853 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.863287 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.863466 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.863686 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:25.864401 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 02:47:25.864636 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21664,"thread_start_us":21497,"threads_started":1}
I20260929 02:47:25.865150 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.834302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.0.97.1:34203)
0929 02:47:25.853787 (+ 19485us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:25.853799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:25.854577 (+ 778us) server_negotiation.cc:407] Connection header received
0929 02:47:25.854691 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:25.854699 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:25.854994 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:25.855201 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:47:25.856200 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.857903 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.860840 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.861850 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.861993 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.862240 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:25.864637 (+ 2397us) server_negotiation.cc:299] Negotiation successful
0929 02:47:25.864817 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19249,"thread_start_us":19084,"threads_started":1}
I20260929 02:47:25.864404 11193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.836276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39233 (local address 127.0.0.1:34442)
0929 02:47:25.854578 (+ 18302us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:25.854600 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:25.854699 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:25.855924 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:25.855931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:25.855952 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:47:25.856346 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.856359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.858232 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.858237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.861115 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.861132 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.863471 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.863477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.863685 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.863727 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:47:25.863836 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 02:47:25.863971 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18189,"mutex_wait_us":108,"thread_start_us":18038,"threads_started":1}
I20260929 02:47:25.865439 11189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:25.833718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42943 (local address 127.0.0.1:60938)
0929 02:47:25.848300 (+ 14582us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:25.848378 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:25.848682 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:25.856599 (+ 7917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:25.856605 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:25.856636 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:47:25.857006 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.857019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.859157 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.859162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.861679 (+ 2517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:25.861692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:25.863544 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:25.863554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:25.863745 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:25.863775 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:47:25.863920 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 02:47:25.864056 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14208,"mutex_wait_us":48,"thread_start_us":15209,"threads_started":1}
I20260929 02:47:25.869179 10844 tablet_service.cc:1467] Processing CreateTablet for tablet 36a4909f2836445c915edb955921653d (DEFAULT_TABLE table=test-table [id=c1351a4cca9f4d0888ddd9feb0a9ddfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:47:25.869194 10978 tablet_service.cc:1467] Processing CreateTablet for tablet 36a4909f2836445c915edb955921653d (DEFAULT_TABLE table=test-table [id=c1351a4cca9f4d0888ddd9feb0a9ddfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:47:25.869854 11112 tablet_service.cc:1467] Processing CreateTablet for tablet 36a4909f2836445c915edb955921653d (DEFAULT_TABLE table=test-table [id=c1351a4cca9f4d0888ddd9feb0a9ddfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:47:25.870543 10844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a4909f2836445c915edb955921653d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:47:25.870541 10978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a4909f2836445c915edb955921653d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:47:25.871085 11112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36a4909f2836445c915edb955921653d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:47:25.882810 11195 tablet_bootstrap.cc:492] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Bootstrap starting.
I20260929 02:47:25.885175 11196 tablet_bootstrap.cc:492] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Bootstrap starting.
I20260929 02:47:25.886830 11197 tablet_bootstrap.cc:492] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Bootstrap starting.
I20260929 02:47:25.886844 11195 tablet_bootstrap.cc:654] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Neither blocks nor log segments found. Creating new log.
I20260929 02:47:25.888315 11195 log.cc:824] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:47:25.889114 11196 tablet_bootstrap.cc:654] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Neither blocks nor log segments found. Creating new log.
I20260929 02:47:25.889858 11197 tablet_bootstrap.cc:654] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Neither blocks nor log segments found. Creating new log.
I20260929 02:47:25.890460 11196 log.cc:824] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:47:25.890957 11197 log.cc:824] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Log is configured to *not* fsync() on all Append() calls
I20260929 02:47:25.892527 11195 tablet_bootstrap.cc:492] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: No bootstrap required, opened a new log
I20260929 02:47:25.892858 11195 ts_tablet_manager.cc:1397] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260929 02:47:25.893487 11196 tablet_bootstrap.cc:492] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: No bootstrap required, opened a new log
I20260929 02:47:25.893740 11196 ts_tablet_manager.cc:1397] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260929 02:47:25.893970 11197 tablet_bootstrap.cc:492] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: No bootstrap required, opened a new log
I20260929 02:47:25.894217 11197 ts_tablet_manager.cc:1397] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260929 02:47:25.902217 11195 raft_consensus.cc:348] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:25.902662 11195 raft_consensus.cc:374] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:47:25.902808 11195 raft_consensus.cc:729] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c575191b4b4c00869247e9a88a9fc9, State: Initialized, Role: FOLLOWER
I20260929 02:47:25.903503 11195 consensus_queue.cc:260] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:25.904680 11197 raft_consensus.cc:348] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:25.904680 11196 raft_consensus.cc:348] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:25.905120 11196 raft_consensus.cc:374] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:47:25.905148 11197 raft_consensus.cc:374] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:47:25.905246 11196 raft_consensus.cc:729] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99530c59667e4bae9a3f9f353c0e018f, State: Initialized, Role: FOLLOWER
I20260929 02:47:25.905237 11197 raft_consensus.cc:729] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fc499bd97494cd1b6582c47677ae744, State: Initialized, Role: FOLLOWER
I20260929 02:47:25.905519 11195 ts_tablet_manager.cc:1428] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260929 02:47:25.905965 11196 consensus_queue.cc:260] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:25.905956 11197 consensus_queue.cc:260] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:25.907121 11177 heartbeater.cc:503] Master 127.0.97.62:38617 was elected leader, sending a full tablet report...
I20260929 02:47:25.908238 11196 ts_tablet_manager.cc:1428] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 02:47:25.909713 11043 heartbeater.cc:503] Master 127.0.97.62:38617 was elected leader, sending a full tablet report...
I20260929 02:47:25.910535 11197 ts_tablet_manager.cc:1428] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Time spent starting tablet: real 0.016s user 0.013s sys 0.004s
W20260929 02:47:25.973378 10910 tablet.cc:2369] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:47:26.004243 11044 tablet.cc:2369] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:47:26.011072 11178 tablet.cc:2369] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:47:26.064508 11202 raft_consensus.cc:482] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:47:26.064963 11202 raft_consensus.cc:504] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:26.067044 11202 leader_election.cc:288] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 80c575191b4b4c00869247e9a88a9fc9 (127.0.97.1:34203), 9fc499bd97494cd1b6582c47677ae744 (127.0.97.2:39233)
I20260929 02:47:26.073283 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.067408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48005 (local address 127.0.97.1:34203)
0929 02:47:26.067614 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:26.067621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:26.068350 (+ 729us) server_negotiation.cc:407] Connection header received
0929 02:47:26.068604 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:26.068607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:26.068665 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:26.068761 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:47:26.069246 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.069952 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.071584 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.072374 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.072540 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.072659 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:26.072920 (+ 261us) server_negotiation.cc:299] Negotiation successful
0929 02:47:26.073042 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:47:26.073276 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.067859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50175 (local address 127.0.97.2:39233)
0929 02:47:26.068061 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:26.068066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:26.068082 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:47:26.068145 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:26.068149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:26.068213 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:26.068330 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:47:26.068839 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.069595 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.071333 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.072181 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.072346 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.072456 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:26.072853 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 02:47:26.072988 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:47:26.073308 11179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.067388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:39233 (local address 127.0.97.3:50175)
0929 02:47:26.067619 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:26.067641 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:26.067835 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:26.068367 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:26.068372 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:26.068428 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:47:26.068675 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:26.068684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.069750 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.069756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:26.071162 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:26.071173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.072356 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.072361 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:26.072661 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.072692 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:47:26.072826 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:47:26.072979 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"spinlock_wait_cycles":4864}
I20260929 02:47:26.073458 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.067396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:34203 (local address 127.0.97.3:48005)
0929 02:47:26.068238 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0929 02:47:26.068252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:47:26.068431 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:47:26.068771 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:47:26.068776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:47:26.068796 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:47:26.069100 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:26.069108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.070100 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.070105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:26.071436 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:47:26.071446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.072610 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.072616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:47:26.072765 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.072787 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:47:26.072894 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 02:47:26.073019 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"mutex_wait_us":65,"spinlock_wait_cycles":25600,"thread_start_us":459,"threads_started":1}
I20260929 02:47:26.074476 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a4909f2836445c915edb955921653d" candidate_uuid: "99530c59667e4bae9a3f9f353c0e018f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fc499bd97494cd1b6582c47677ae744" is_pre_election: true
I20260929 02:47:26.074476 10864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a4909f2836445c915edb955921653d" candidate_uuid: "99530c59667e4bae9a3f9f353c0e018f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c575191b4b4c00869247e9a88a9fc9" is_pre_election: true
I20260929 02:47:26.075057 10864 raft_consensus.cc:2454] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99530c59667e4bae9a3f9f353c0e018f in term 0.
I20260929 02:47:26.075057 10998 raft_consensus.cc:2454] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99530c59667e4bae9a3f9f353c0e018f in term 0.
I20260929 02:47:26.075784 11067 leader_election.cc:302] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 80c575191b4b4c00869247e9a88a9fc9, 99530c59667e4bae9a3f9f353c0e018f, 9fc499bd97494cd1b6582c47677ae744; no voters:
I20260929 02:47:26.076197 11202 raft_consensus.cc:2781] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:47:26.076344 11202 raft_consensus.cc:482] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:47:26.076449 11202 raft_consensus.cc:3037] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:47:26.078656 11202 raft_consensus.cc:504] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:26.079241 11202 leader_election.cc:288] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [CANDIDATE]: Term 1 election: Requested vote from peers 80c575191b4b4c00869247e9a88a9fc9 (127.0.97.1:34203), 9fc499bd97494cd1b6582c47677ae744 (127.0.97.2:39233)
I20260929 02:47:26.079644 10864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a4909f2836445c915edb955921653d" candidate_uuid: "99530c59667e4bae9a3f9f353c0e018f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80c575191b4b4c00869247e9a88a9fc9"
I20260929 02:47:26.079674 10998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36a4909f2836445c915edb955921653d" candidate_uuid: "99530c59667e4bae9a3f9f353c0e018f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fc499bd97494cd1b6582c47677ae744"
I20260929 02:47:26.079865 10864 raft_consensus.cc:3037] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:47:26.079871 10998 raft_consensus.cc:3037] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:47:26.082175 10864 raft_consensus.cc:2454] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99530c59667e4bae9a3f9f353c0e018f in term 1.
I20260929 02:47:26.082409 10998 raft_consensus.cc:2454] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99530c59667e4bae9a3f9f353c0e018f in term 1.
I20260929 02:47:26.082713 11068 leader_election.cc:302] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c575191b4b4c00869247e9a88a9fc9, 99530c59667e4bae9a3f9f353c0e018f; no voters:
I20260929 02:47:26.083094 11202 raft_consensus.cc:2781] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:47:26.083554 11202 raft_consensus.cc:686] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 1 LEADER]: Becoming Leader. State: Replica: 99530c59667e4bae9a3f9f353c0e018f, State: Running, Role: LEADER
I20260929 02:47:26.084061 11202 consensus_queue.cc:237] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } }
I20260929 02:47:26.088941 10720 catalog_manager.cc:5495] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f reported cstate change: term changed from 0 to 1, leader changed from <none> to 99530c59667e4bae9a3f9f353c0e018f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "99530c59667e4bae9a3f9f353c0e018f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99530c59667e4bae9a3f9f353c0e018f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:47:26.157123 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.152053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.0.97.3:42943)
0929 02:47:26.152289 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:26.152295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:26.152315 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:47:26.152452 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:26.152456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:26.152529 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:26.152718 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:47:26.153097 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.153809 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.155111 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.155510 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.155644 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.156114 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:47:26.156640 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:47:26.156822 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:26.156871 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 02:47:26.156932 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 02:47:26.164889 10864 raft_consensus.cc:1261] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 1 FOLLOWER]: Refusing update from remote peer 99530c59667e4bae9a3f9f353c0e018f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:47:26.164947 10998 raft_consensus.cc:1261] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 1 FOLLOWER]: Refusing update from remote peer 99530c59667e4bae9a3f9f353c0e018f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:47:26.165823 11205 consensus_queue.cc:1035] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [LEADER]: Connected to new peer: Peer: permanent_uuid: "80c575191b4b4c00869247e9a88a9fc9" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 34203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:47:26.166168 11202 consensus_queue.cc:1035] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fc499bd97494cd1b6582c47677ae744" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 39233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:47:26.175209 11208 mvcc.cc:205] Tried to move back new op lower bound from 7334502589082034176 to 7334502588764614656. Current Snapshot: MvccSnapshot[applied={T|T < 7334502589082034176}]
I20260929 02:47:26.178699 11209 mvcc.cc:205] Tried to move back new op lower bound from 7334502589082034176 to 7334502588764614656. Current Snapshot: MvccSnapshot[applied={T|T < 7334502589082034176}]
I20260929 02:47:26.193276 10720 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:47:26.193563 10720 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:47:26.197499 10720 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46ea770019c14409b66e8fb5f4bc2fb3: Sending DeleteTablet for 3 replicas of tablet 36a4909f2836445c915edb955921653d
I20260929 02:47:26.198686 10844 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a4909f2836445c915edb955921653d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:47:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36576
I20260929 02:47:26.198599 11112 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a4909f2836445c915edb955921653d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:47:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60938
I20260929 02:47:26.199865 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10781
I20260929 02:47:26.199852 10978 tablet_service.cc:1514] Processing DeleteTablet for tablet 36a4909f2836445c915edb955921653d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:47:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34442
I20260929 02:47:26.209816 11219 tablet_replica.cc:330] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: stopping tablet replica
I20260929 02:47:26.210537 11219 raft_consensus.cc:2229] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:47:26.211261 11219 raft_consensus.cc:2258] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:47:26.215058 11219 ts_tablet_manager.cc:1905] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:47:26.215477 11217 tablet_replica.cc:330] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: stopping tablet replica
I20260929 02:47:26.215996 11217 raft_consensus.cc:2229] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:47:26.216609 11217 raft_consensus.cc:2258] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:47:26.219017 11217 ts_tablet_manager.cc:1905] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:47:26.222496 11219 ts_tablet_manager.cc:1918] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:47:26.222669 11219 log.cc:1196] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/36a4909f2836445c915edb955921653d
I20260929 02:47:26.223272 11219 ts_tablet_manager.cc:1939] T 36a4909f2836445c915edb955921653d P 9fc499bd97494cd1b6582c47677ae744: Deleting consensus metadata
Sep 29 02:47:26 dist-test-slave-6tjh krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:47:26.225870 10706 catalog_manager.cc:4841] TS 9fc499bd97494cd1b6582c47677ae744 (127.0.97.2:39233): tablet 36a4909f2836445c915edb955921653d (table test-table [id=c1351a4cca9f4d0888ddd9feb0a9ddfb]) successfully deleted
I20260929 02:47:26.226284 11217 ts_tablet_manager.cc:1918] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:47:26.226445 11217 log.cc:1196] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/36a4909f2836445c915edb955921653d
I20260929 02:47:26.226869 11217 ts_tablet_manager.cc:1939] T 36a4909f2836445c915edb955921653d P 80c575191b4b4c00869247e9a88a9fc9: Deleting consensus metadata
I20260929 02:47:26.227316 11218 tablet_replica.cc:330] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: stopping tablet replica
I20260929 02:47:26.227813 11218 raft_consensus.cc:2229] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:47:26.228598 11218 raft_consensus.cc:2258] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:47:26.228716 10707 catalog_manager.cc:4841] TS 80c575191b4b4c00869247e9a88a9fc9 (127.0.97.1:34203): tablet 36a4909f2836445c915edb955921653d (table test-table [id=c1351a4cca9f4d0888ddd9feb0a9ddfb]) successfully deleted
I20260929 02:47:26.230615 11218 ts_tablet_manager.cc:1905] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:47:26.237347 11192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.201110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36588 (local address 127.0.97.1:34203)
0929 02:47:26.201342 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:26.201350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:26.208617 (+ 7267us) server_negotiation.cc:407] Connection header received
0929 02:47:26.209693 (+ 1076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:26.209709 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:26.209828 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:26.209910 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:26.213309 (+ 3399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.214534 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.218304 (+ 3770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.219004 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.219112 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.230253 (+ 11141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:26.230268 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:26.230285 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:26.230337 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:26.234169 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:26.235066 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:26.235076 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:26.235080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:26.235236 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:26.235655 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:26.235660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:26.235664 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:26.236038 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:26.236342 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:26.236720 (+ 378us) server_negotiation.cc:299] Negotiation successful
0929 02:47:26.236942 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:47:26.238091 10904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:26.243235 11218 ts_tablet_manager.cc:1918] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:47:26.243453 11218 log.cc:1196] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/36a4909f2836445c915edb955921653d
I20260929 02:47:26.244061 11218 ts_tablet_manager.cc:1939] T 36a4909f2836445c915edb955921653d P 99530c59667e4bae9a3f9f353c0e018f: Deleting consensus metadata
I20260929 02:47:26.246420 10704 catalog_manager.cc:4841] TS 99530c59667e4bae9a3f9f353c0e018f (127.0.97.3:42943): tablet 36a4909f2836445c915edb955921653d (table test-table [id=c1351a4cca9f4d0888ddd9feb0a9ddfb]) successfully deleted
I20260929 02:47:26.330166 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10781
I20260929 02:47:26.352813 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10915
Sep 29 02:47:26 dist-test-slave-6tjh krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:47:26.371142 11194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.353717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.0.97.2:39233)
0929 02:47:26.353899 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:26.353904 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:26.353922 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:47:26.354146 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:26.354150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:26.354221 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:26.354305 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:26.356013 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.356841 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.358741 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.359301 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.359406 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.365071 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:26.365087 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:26.365104 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:26.365167 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:26.368348 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:26.369222 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:26.369229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:26.369232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:26.369350 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:26.369767 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:26.369770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:26.369772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:26.370130 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:26.370333 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:26.370689 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 02:47:26.370842 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:47:26.371796 11038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:26.462678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10915
I20260929 02:47:26.479802 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 11049
Sep 29 02:47:26 dist-test-slave-6tjh krb5kdc[10675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:47:26.498843 11191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:47:26.480679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60958 (local address 127.0.97.3:42943)
0929 02:47:26.480841 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 02:47:26.480846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:47:26.480956 (+ 110us) server_negotiation.cc:407] Connection header received
0929 02:47:26.481145 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:47:26.481149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:47:26.481212 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:47:26.481296 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:47:26.483106 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.483944 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.485894 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:47:26.486324 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:47:26.486430 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:47:26.492011 (+ 5581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:47:26.492024 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:47:26.492039 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:47:26.492078 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:47:26.495603 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:26.496512 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:26.496520 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:26.496522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:26.496732 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:47:26.497157 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:47:26.497162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:47:26.497166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:47:26.497557 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:47:26.497887 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 02:47:26.498174 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 02:47:26.498406 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:47:26.499527 11172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:26.617097 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 11049
I20260929 02:47:26.635790 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 10688
I20260929 02:47:26.637072 10750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:47:26.744107 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 10688
2026-09-29T02:47:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4420 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:47:26 dist-test-slave-6tjh krb5kdc[11227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:47:26 dist-test-slave-6tjh krb5kdc[11227](info): set up 2 sockets
Sep 29 02:47:26 dist-test-slave-6tjh krb5kdc[11227](info): commencing operation
krb5kdc: starting...
W20260929 02:47:27.845444 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:47:27 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:47:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 02:47:28.173935 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMdzdaQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/postgres -l logfile start
2026-09-29 02:47:30.776 UTC [11257] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:47:30.776 UTC [11257] LOG: listening on IPv4 address "127.0.97.20", port 58141
2026-09-29 02:47:30.780 UTC [11257] LOG: listening on Unix socket "/tmp/.s.PGSQL.58141"
2026-09-29 02:47:30.805 UTC [11260] LOG: database system was shut down at 2026-09-29 02:47:29 UTC
2026-09-29 02:47:30.808 UTC [11257] LOG: database system is ready to accept connections
I20260929 02:47:31.785353 388 mini_postgres.cc:96] Postgres bound to 58141
2026-09-29 02:47:31.803 UTC [11268] FATAL: database "slave" does not exist
127.0.97.20:58141 - accepting connections
I20260929 02:47:31.804389 388 mini_ranger.cc:161] Starting Ranger...
I20260929 02:47:31.830993 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:47:32.341569 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:47:32.341706 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-admin
2026-09-29 02:47:32,647 [I] DB FLAVOR :POSTGRES
2026-09-29 02:47:32,648 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:47:32,648 [I] Checking connection..
2026-09-29 02:47:32,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:47:32,916 [I] Checking connection passed.
2026-09-29 02:47:32,916 [I] --------- Verifying version history table ---------
2026-09-29 02:47:32,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:47:33,176 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:47:33,177 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:47:33,435 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:47:33,435 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:47:33,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:47:33.690 UTC [11332] WARNING: there is no transaction in progress
2026-09-29 02:47:33,696 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:47:33,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:47:33,939 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:47:33,940 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:47:33,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:47:34,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:47:34,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6tjh.c.gcp-upstream.internal','N') ;"
2026-09-29 02:47:34,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:47:34,963 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:47:34,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:47:35,220 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:47:35,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:47:35,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:47:35,692 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:47:35,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:47:36.688 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.705 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.722 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.738 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.755 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.774 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.831 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.837 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.852 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.862 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.872 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.886 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.897 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.903 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.910 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:36.917 UTC [11440] WARNING: there is no transaction in progress
2026-09-29 02:47:37,600 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:47:37,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:47:37,856 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:47:37,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:47:38,086 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:47:38,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:47:38,330 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:47:38,330 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:47:38,562 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:47:38,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6tjh.c.gcp-upstream.internal';"
2026-09-29 02:47:39,063 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:47:39,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:47:39,307 [I] DB_PATCHES have already been applied
I20260929 02:47:39.313897 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMdzdaQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:47:39.314024 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 02:47:39.328662 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:40189
Sep 29, 2026 2:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:47:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:47:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:47:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:47:40.478657 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.674s user 0.004s sys 0.037s
Sep 29, 2026 2:48:00 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:48:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:48:00 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:48:00 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:48:01 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:48:01.852353 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:48:02 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650082, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:48:02.023684 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:48:02.045065 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:48:02.936663 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:48:02.936827 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms
I20260929 02:48:02.936990 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:58141
2026-09-29 02:48:03,343 [I] DB FLAVOR :POSTGRES
2026-09-29 02:48:03,344 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:48:03,344 [I] Checking connection
2026-09-29 02:48:03,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:48:03,582 [I] connection success
2026-09-29 02:48:03,582 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:48:03,582 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:48:03,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:48:03,858 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:48:03,858 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:48:03,858 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:48:03,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:58141/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:48:04,111 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:48:04.662691 388 mini_ranger_kms.cc:310] Created kms service
I20260929 02:48:04.837122 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:48:04.885664 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:48:04 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650082, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:48:05.279359 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:48:05.279707 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:48:05.279785 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 02:48:05.301793 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11620
I20260929 02:48:05.302001 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:53413
Sep 29, 2026 2:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:48:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:48:07.638902 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.615s user 0.125s sys 0.299s
I20260929 02:48:07.639142 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:48:07.639344 388 mini_ranger_kms.cc:401] 127.0.97.20:53413/kms/v1/keys
Sep 29 02:48:07 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650087, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:48:08 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650087, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 02:48:08 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650087, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 02:48:10 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:48:10.858078 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34269
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34269
--ranger_config_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53413/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:48:11.200007 11671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:48:11.200843 11671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:48:11.201010 11671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:48:11.208196 11671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:48:11.208349 11671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:48:11.208455 11671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:48:11.208581 11671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:48:11.208652 11671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:48:11.219498 11671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:53413/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34269
--ranger_config_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:48:11.220914 11671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:48:11.222347 11671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:48:11.235106 11676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:11.235085 11679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:11.235293 11677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:48:11.238057 11671 server_base.cc:1034] running on GCE node
I20260929 02:48:11.238925 11671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:48:11.240341 11671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:48:11.241561 11671 hybrid_clock.cc:648] HybridClock initialized: now 1790650091241516 us; error 46 us; skew 500 ppm
I20260929 02:48:11.241894 11671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:48:11 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:48:11.249946 11671 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:48:11.251492 11671 webserver.cc:460] Webserver started at http://127.0.97.62:35741/ using document root <none> and password file <none>
I20260929 02:48:11.252147 11671 fs_manager.cc:359] Metadata directory not provided
I20260929 02:48:11.252235 11671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:48:11.252489 11671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:48:11 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:48:11.307144 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "29c52a94a1064f00947f9e94710ac433"
format_stamp: "Formatted at 2026-09-29 02:48:11 on dist-test-slave-6tjh"
server_key: "17c66fac5623aaf0ed88e64d29a5a7b4"
server_key_iv: "d0c1c777c1af8b28b7c053a9e63a1860"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:11.308290 11671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "29c52a94a1064f00947f9e94710ac433"
format_stamp: "Formatted at 2026-09-29 02:48:11 on dist-test-slave-6tjh"
server_key: "17c66fac5623aaf0ed88e64d29a5a7b4"
server_key_iv: "d0c1c777c1af8b28b7c053a9e63a1860"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:11.315557 11671 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260929 02:48:11.339388 11686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:11.341400 11671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 02:48:11.341643 11671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "29c52a94a1064f00947f9e94710ac433"
format_stamp: "Formatted at 2026-09-29 02:48:11 on dist-test-slave-6tjh"
server_key: "17c66fac5623aaf0ed88e64d29a5a7b4"
server_key_iv: "d0c1c777c1af8b28b7c053a9e63a1860"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:11.341910 11671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:48:11.362377 11671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:48:11.367316 11671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:48:11.367647 11671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:48:11.386731 11671 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34269
I20260929 02:48:11.386761 11737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34269 every 8 connection(s)
I20260929 02:48:11.388198 11671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:48:11.391985 11738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:48:11.398276 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 11671
I20260929 02:48:11.398532 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:48:11.399153 388 external_mini_cluster.cc:1428] Setting key 3dec45867c0980dac7a2cc67038f8d9e
I20260929 02:48:11.405930 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433: Bootstrap starting.
I20260929 02:48:11.409827 11738 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433: Neither blocks nor log segments found. Creating new log.
I20260929 02:48:11.411427 11738 log.cc:824] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433: Log is configured to *not* fsync() on all Append() calls
I20260929 02:48:11.414386 11738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433: No bootstrap required, opened a new log
Sep 29 02:48:11 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:48:11.422410 11738 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c52a94a1064f00947f9e94710ac433" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34269 } }
I20260929 02:48:11.422917 11738 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:48:11.423041 11738 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c52a94a1064f00947f9e94710ac433, State: Initialized, Role: FOLLOWER
I20260929 02:48:11.423966 11741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:11.401284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60720 (local address 127.0.97.62:34269)
0929 02:48:11.402252 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:11.402268 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:11.402325 (+ 57us) server_negotiation.cc:407] Connection header received
0929 02:48:11.404788 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:11.404818 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:11.405359 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:11.405839 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:11.407105 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:11.409186 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:11.410527 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:11.412118 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:11.412363 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:11.416144 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:11.416152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:11.416170 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:11.416208 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:11.419303 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:11.420009 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:11.420019 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:11.420026 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:11.420175 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:11.420591 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:11.420596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:11.420601 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:11.421152 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:11.421477 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:11.421902 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 02:48:11.422304 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":443,"threads_started":1}
I20260929 02:48:11.424047 11738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c52a94a1064f00947f9e94710ac433" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34269 } }
I20260929 02:48:11.424338 11738 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:48:11.424507 11738 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:48:11.424734 11738 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:48:11.427762 11738 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c52a94a1064f00947f9e94710ac433" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34269 } }
I20260929 02:48:11.428241 11738 leader_election.cc:302] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29c52a94a1064f00947f9e94710ac433; no voters:
I20260929 02:48:11.428881 11738 leader_election.cc:288] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:48:11.429067 11743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:48:11.429804 11743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [term 1 LEADER]: Becoming Leader. State: Replica: 29c52a94a1064f00947f9e94710ac433, State: Running, Role: LEADER
I20260929 02:48:11.430390 11743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c52a94a1064f00947f9e94710ac433" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34269 } }
I20260929 02:48:11.431092 11738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:48:11.433465 11744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29c52a94a1064f00947f9e94710ac433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c52a94a1064f00947f9e94710ac433" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34269 } } }
I20260929 02:48:11.433583 11745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c52a94a1064f00947f9e94710ac433. Latest consensus state: current_term: 1 leader_uuid: "29c52a94a1064f00947f9e94710ac433" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29c52a94a1064f00947f9e94710ac433" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34269 } } }
I20260929 02:48:11.433981 11745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [sys.catalog]: This master's current role is: LEADER
I20260929 02:48:11.434094 11744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433 [sys.catalog]: This master's current role is: LEADER
I20260929 02:48:11.440452 11738 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:48:11.440755 11751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:48:11.455716 11751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:48:11.468683 11751 catalog_manager.cc:1347] Generated new cluster ID: f162a0acf486403fbb2cfd75e60c69b0
I20260929 02:48:11.468812 11751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:48:11.493445 11751 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:48:11.495586 11751 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:48:11.512699 11751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433: Generated new TSK 0
I20260929 02:48:11.513855 11751 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:48:12.298382 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34269
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:48:12 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:48:12.782634 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:48:12.783138 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:48:12.783355 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:48:12.795542 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:48:12.795778 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:48:12.795981 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:48:12.812363 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:53413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:48:12.814253 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:48:12.816035 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:48:12.833475 11793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:12.834846 11794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:12.846498 11796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:48:12.846760 11783 server_base.cc:1034] running on GCE node
I20260929 02:48:12.847635 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:48:12.849076 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:48:12.850400 11783 hybrid_clock.cc:648] HybridClock initialized: now 1790650092850331 us; error 64 us; skew 500 ppm
I20260929 02:48:12.850793 11783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:48:12 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:48:12.855072 11783 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:48:12.857121 11783 webserver.cc:460] Webserver started at http://127.0.97.1:39339/ using document root <none> and password file <none>
I20260929 02:48:12.857915 11783 fs_manager.cc:359] Metadata directory not provided
I20260929 02:48:12.858058 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:48:12.858404 11783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:48:12 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:48:12.879916 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "2b85fa482037452ba59e4cbd7654836f"
format_stamp: "Formatted at 2026-09-29 02:48:12 on dist-test-slave-6tjh"
server_key: "ecbe2a48e6ae2fefdb4711129ffc62c3"
server_key_iv: "2187e76ecc272582e5f22582982eac4e"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:12.881285 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2b85fa482037452ba59e4cbd7654836f"
format_stamp: "Formatted at 2026-09-29 02:48:12 on dist-test-slave-6tjh"
server_key: "ecbe2a48e6ae2fefdb4711129ffc62c3"
server_key_iv: "2187e76ecc272582e5f22582982eac4e"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:12.892699 11783 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.007s sys 0.001s
I20260929 02:48:12.911983 11803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:12.914160 11783 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 02:48:12.914383 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "2b85fa482037452ba59e4cbd7654836f"
format_stamp: "Formatted at 2026-09-29 02:48:12 on dist-test-slave-6tjh"
server_key: "ecbe2a48e6ae2fefdb4711129ffc62c3"
server_key_iv: "2187e76ecc272582e5f22582982eac4e"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:12.914702 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:48:12.939909 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:48:12.945421 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:48:12.945823 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:48:12.953473 11783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:48:12.969528 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:48:12.969724 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 02:48:12.970134 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:48:12.970219 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:48:13 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:48:13.353209 11783 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44883
I20260929 02:48:13.353228 11915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44883 every 8 connection(s)
I20260929 02:48:13.354660 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:48:13.364957 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 11783
I20260929 02:48:13.365188 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:48:13.366081 388 external_mini_cluster.cc:1428] Setting key c6940062cc8405c5f16d3b38b5d648e9
Sep 29 02:48:13 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:48:13.398893 11918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:13.357846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59127 (local address 127.0.97.62:34269)
0929 02:48:13.358334 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:13.358341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:13.359467 (+ 1126us) server_negotiation.cc:407] Connection header received
0929 02:48:13.362088 (+ 2621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:13.362093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:13.362166 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:13.362270 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:13.364413 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:13.365799 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:13.369522 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:13.370425 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:13.370593 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:13.379549 (+ 8956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:13.379557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:13.379562 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:13.379608 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:13.385013 (+ 5405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:13.386050 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:13.386056 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:13.386060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:13.394995 (+ 8935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:13.396354 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:13.396362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:13.396366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:13.396823 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:13.396983 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:13.398252 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0929 02:48:13.398519 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20260929 02:48:13.401667 11919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:13.358470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34269 (local address 127.0.97.1:59127)
0929 02:48:13.359267 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:13.359323 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:13.361690 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:13.362587 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:13.362599 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:13.363266 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:48:13.364117 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:13.364143 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:13.366046 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:13.366052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:13.368904 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:13.368939 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:13.370711 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:13.370720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:13.371086 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:13.372413 (+ 1327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:48:13.372443 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:48:13.379189 (+ 6746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:48:13.385212 (+ 6023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:48:13.385222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:48:13.385242 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:48:13.385873 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:48:13.395239 (+ 9366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:48:13.395246 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:48:13.395251 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:48:13.396157 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:48:13.397094 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:48:13.397104 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:48:13.397676 (+ 572us) client_negotiation.cc:769] Sending connection context
0929 02:48:13.399483 (+ 1807us) client_negotiation.cc:240] Negotiation successful
0929 02:48:13.400103 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":308,"threads_started":1}
I20260929 02:48:13.404003 11916 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34269
I20260929 02:48:13.404392 11916 heartbeater.cc:464] Registering TS with master...
I20260929 02:48:13.405397 11916 heartbeater.cc:511] Master 127.0.97.62:34269 requested a full tablet report, sending...
I20260929 02:48:13.408272 11703 ts_manager.cc:194] Registered new tserver with Master: 2b85fa482037452ba59e4cbd7654836f (127.0.97.1:44883)
I20260929 02:48:13.410032 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:48:13.436928 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34269
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:48:13 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 02:48:13.912164 11922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:48:13.912448 11922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:48:13.912629 11922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:48:13.920804 11922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:48:13.920920 11922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:48:13.921120 11922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:48:13.930951 11922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:53413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:48:13.932323 11922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:48:13.933646 11922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:48:13.948194 11937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:13.948472 11939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:13.948217 11936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:48:13.950050 11922 server_base.cc:1034] running on GCE node
I20260929 02:48:13.950659 11922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:48:13.951692 11922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:48:13.952898 11922 hybrid_clock.cc:648] HybridClock initialized: now 1790650093952861 us; error 53 us; skew 500 ppm
I20260929 02:48:13.953210 11922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:48:13 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:48:13.956887 11922 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:48:13.958467 11922 webserver.cc:460] Webserver started at http://127.0.97.2:43363/ using document root <none> and password file <none>
I20260929 02:48:13.959157 11922 fs_manager.cc:359] Metadata directory not provided
I20260929 02:48:13.959301 11922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:48:13.959626 11922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:48:13 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:48:13.978277 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "a9ce1467203a4bb8b9720d350cda14ff"
format_stamp: "Formatted at 2026-09-29 02:48:13 on dist-test-slave-6tjh"
server_key: "b8c78a759e4052ddff264470cf6c26d6"
server_key_iv: "df10a8bfc0f8f5d9adf7bf13b44ce91f"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:13.979426 11922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a9ce1467203a4bb8b9720d350cda14ff"
format_stamp: "Formatted at 2026-09-29 02:48:13 on dist-test-slave-6tjh"
server_key: "b8c78a759e4052ddff264470cf6c26d6"
server_key_iv: "df10a8bfc0f8f5d9adf7bf13b44ce91f"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:13.991425 11922 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.003s
I20260929 02:48:14.009291 11946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:14.011286 11922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:48:14.011485 11922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "a9ce1467203a4bb8b9720d350cda14ff"
format_stamp: "Formatted at 2026-09-29 02:48:13 on dist-test-slave-6tjh"
server_key: "b8c78a759e4052ddff264470cf6c26d6"
server_key_iv: "df10a8bfc0f8f5d9adf7bf13b44ce91f"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:14.011826 11922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:48:14.044574 11922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:48:14.050294 11922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:48:14.050639 11922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:48:14.051937 11922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:48:14.053995 11922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:48:14.054108 11922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:14.054244 11922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:48:14.054328 11922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:14.098623 11922 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36399
I20260929 02:48:14.098680 12058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36399 every 8 connection(s)
I20260929 02:48:14.100289 11922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:48:14.103637 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 11922
I20260929 02:48:14.103816 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:48:14.104607 388 external_mini_cluster.cc:1428] Setting key 92eda05fb46a78f7d50c6e5ae5460cfc
Sep 29 02:48:14 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:48:14.125893 12061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.103226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38357 (local address 127.0.97.62:34269)
0929 02:48:14.108968 (+ 5742us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:14.108975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:14.108998 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:48:14.110186 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:14.110192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:14.110284 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:14.110394 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:14.111883 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.112688 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.114268 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.114685 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.114820 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.118678 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:14.118683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:14.118686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:14.118723 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:14.122107 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:14.123119 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:14.123125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:14.123129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:14.123283 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:14.123840 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:14.123845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:14.123849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:14.124191 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:14.124339 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:14.125287 (+ 948us) server_negotiation.cc:299] Negotiation successful
0929 02:48:14.125517 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5597,"thread_start_us":5469,"threads_started":1}
I20260929 02:48:14.127307 12062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.103781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34269 (local address 127.0.97.2:38357)
0929 02:48:14.107633 (+ 3852us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:14.107678 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:14.109702 (+ 2024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:14.110661 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:14.110670 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:14.111127 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:48:14.111709 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.111727 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.112842 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.112848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.114101 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.114120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.114912 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.114919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.115162 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.115865 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:48:14.115891 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:48:14.118396 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:48:14.122279 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:48:14.122293 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:48:14.122314 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:48:14.122978 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:48:14.123412 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:48:14.123420 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:48:14.123427 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:48:14.123711 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:48:14.124333 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:48:14.124345 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:48:14.124835 (+ 490us) client_negotiation.cc:769] Sending connection context
0929 02:48:14.125173 (+ 338us) client_negotiation.cc:240] Negotiation successful
0929 02:48:14.125757 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2265,"thread_start_us":2053,"threads_started":1}
I20260929 02:48:14.128976 12059 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34269
I20260929 02:48:14.129372 12059 heartbeater.cc:464] Registering TS with master...
I20260929 02:48:14.130257 12059 heartbeater.cc:511] Master 127.0.97.62:34269 requested a full tablet report, sending...
I20260929 02:48:14.132519 11703 ts_manager.cc:194] Registered new tserver with Master: a9ce1467203a4bb8b9720d350cda14ff (127.0.97.2:36399)
I20260929 02:48:14.133778 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38357
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:48:14.165745 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:53413/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34269
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:48:14.418282 11916 heartbeater.cc:503] Master 127.0.97.62:34269 was elected leader, sending a full tablet report...
W20260929 02:48:14.514546 12065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:48:14.514884 12065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:48:14.515057 12065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:48:14.522543 12065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:48:14.522702 12065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:48:14.522895 12065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:48:14.533273 12065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:53413/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:48:14.534586 12065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:48:14.535914 12065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:48:14.550066 12070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:14.550081 12071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:48:14.549993 12073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:48:14.551082 12065 server_base.cc:1034] running on GCE node
I20260929 02:48:14.551700 12065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:48:14.552753 12065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:48:14.553977 12065 hybrid_clock.cc:648] HybridClock initialized: now 1790650094553934 us; error 30 us; skew 500 ppm
I20260929 02:48:14.554354 12065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:48:14 dist-test-slave-6tjh krb5kdc[11227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:48:14.557611 12065 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:48:14.559154 12065 webserver.cc:460] Webserver started at http://127.0.97.3:46175/ using document root <none> and password file <none>
I20260929 02:48:14.559851 12065 fs_manager.cc:359] Metadata directory not provided
I20260929 02:48:14.559994 12065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:48:14.560312 12065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:48:14 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:48:14.574237 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "91e3154d95f741e5958a38f87afcae2e"
format_stamp: "Formatted at 2026-09-29 02:48:14 on dist-test-slave-6tjh"
server_key: "094c1e77410f530c68f980f6e7533cfe"
server_key_iv: "6f0db04b206cb23d791a5e7a2a51b269"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:14.575156 12065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "91e3154d95f741e5958a38f87afcae2e"
format_stamp: "Formatted at 2026-09-29 02:48:14 on dist-test-slave-6tjh"
server_key: "094c1e77410f530c68f980f6e7533cfe"
server_key_iv: "6f0db04b206cb23d791a5e7a2a51b269"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:14.580951 12065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:48:14.598261 12080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:14.600198 12065 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:48:14.600397 12065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "91e3154d95f741e5958a38f87afcae2e"
format_stamp: "Formatted at 2026-09-29 02:48:14 on dist-test-slave-6tjh"
server_key: "094c1e77410f530c68f980f6e7533cfe"
server_key_iv: "6f0db04b206cb23d791a5e7a2a51b269"
server_key_version: "kuduclusterkey@0"
I20260929 02:48:14.600689 12065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:48:14.620352 12065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:48:14.624147 12065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:48:14.624439 12065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:48:14.625785 12065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:48:14.627152 12065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:48:14.627223 12065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:14.627383 12065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:48:14.627432 12065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:48:14.661492 12065 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38407
I20260929 02:48:14.661511 12192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38407 every 8 connection(s)
I20260929 02:48:14.662781 12065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:48:14.669914 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 12065
I20260929 02:48:14.670115 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:48:14.670537 388 external_mini_cluster.cc:1428] Setting key 2366345d6b25792642d3aadccd7916d4
Sep 29 02:48:14 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:48:14.682655 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.665345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51307 (local address 127.0.97.62:34269)
0929 02:48:14.665753 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:14.665757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:14.666821 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 02:48:14.668583 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:14.668589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:14.668696 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:14.668817 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:14.670355 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.671199 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.673770 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.674231 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.674328 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.677438 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:14.677442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:14.677446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:14.677477 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:14.680082 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:14.680775 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:14.680779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:14.680781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:14.680920 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:14.681295 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:14.681298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:14.681300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:14.681552 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:14.681656 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:14.682255 (+ 599us) server_negotiation.cc:299] Negotiation successful
0929 02:48:14.682403 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":210,"threads_started":1}
I20260929 02:48:14.683669 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.665940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34269 (local address 127.0.97.3:51307)
0929 02:48:14.666658 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:14.666700 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:14.668280 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:14.669046 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:14.669054 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:14.669526 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:48:14.670165 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.670203 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.671382 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.671388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.673616 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.673642 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.674413 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.674418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.674628 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.675237 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:48:14.675259 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:48:14.677254 (+ 1995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:48:14.680194 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:48:14.680199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:48:14.680212 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:48:14.680678 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:48:14.681010 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:48:14.681013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:48:14.681015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:48:14.681187 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:48:14.681657 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:48:14.681663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:48:14.681937 (+ 274us) client_negotiation.cc:769] Sending connection context
0929 02:48:14.682169 (+ 232us) client_negotiation.cc:240] Negotiation successful
0929 02:48:14.682512 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":315,"threads_started":1}
I20260929 02:48:14.685083 12193 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34269
I20260929 02:48:14.685385 12193 heartbeater.cc:464] Registering TS with master...
I20260929 02:48:14.686096 12193 heartbeater.cc:511] Master 127.0.97.62:34269 requested a full tablet report, sending...
I20260929 02:48:14.687395 11703 ts_manager.cc:194] Registered new tserver with Master: 91e3154d95f741e5958a38f87afcae2e (127.0.97.3:38407)
I20260929 02:48:14.688171 11703 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51307
I20260929 02:48:14.698367 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:48:14.715140 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.702758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.0.97.62:34269)
0929 02:48:14.703353 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:14.703363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:14.703744 (+ 381us) server_negotiation.cc:407] Connection header received
0929 02:48:14.703862 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:14.703865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:14.703943 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:14.704020 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:14.705447 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.706127 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.707739 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.708130 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.708213 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.710406 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:14.710411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:14.710414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:14.710442 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:14.712998 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:14.713544 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:14.713547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:14.713550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:14.713654 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:14.713955 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:14.713958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:14.713959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:14.714199 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:14.714393 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:14.714642 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 02:48:14.714790 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:48:14.719391 11703 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41962:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:48:14.726989 11703 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:48:14.765483 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.740018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.0.97.3:38407)
0929 02:48:14.757034 (+ 17016us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:14.757047 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:14.757091 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:48:14.757212 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:14.757223 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:14.757543 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:14.757803 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:48:14.758692 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.760233 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.762490 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.764046 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.764181 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.764407 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:14.764950 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 02:48:14.765119 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16722,"thread_start_us":16579,"threads_started":1}
I20260929 02:48:14.765501 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.739835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38407 (local address 127.0.0.1:44748)
0929 02:48:14.754564 (+ 14729us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:14.754625 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:14.754898 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:14.757794 (+ 2896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:14.757798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:14.757822 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:48:14.758410 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.758433 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.760401 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.760406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.762333 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.762345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.764422 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.764430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.764668 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.764738 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 02:48:14.764883 (+ 145us) client_negotiation.cc:240] Negotiation successful
0929 02:48:14.765068 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14341,"mutex_wait_us":65,"thread_start_us":15319,"threads_started":1}
I20260929 02:48:14.766090 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.740117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44883 (local address 127.0.0.1:59452)
0929 02:48:14.755532 (+ 15415us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:14.755552 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:14.756260 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:14.757674 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:14.757679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:14.757702 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:48:14.758457 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.758476 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.760345 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.760356 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.762845 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.762858 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.765350 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.765356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.765549 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.765582 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:48:14.765670 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:48:14.765800 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15295,"mutex_wait_us":82,"thread_start_us":18251,"threads_started":1}
I20260929 02:48:14.766746 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.740737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59452 (local address 127.0.97.1:44883)
0929 02:48:14.751351 (+ 10614us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:14.751362 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:14.755660 (+ 4298us) server_negotiation.cc:407] Connection header received
0929 02:48:14.757045 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:14.757053 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:14.757270 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:14.757431 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:48:14.758702 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.760206 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.763037 (+ 2831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.764904 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.765010 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.765212 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:14.766393 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0929 02:48:14.766515 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10389,"thread_start_us":10214,"threads_started":1}
I20260929 02:48:14.768204 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.740439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36399 (local address 127.0.0.1:42712)
0929 02:48:14.758644 (+ 18205us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:14.758665 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:14.758798 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:14.760069 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:14.760074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:14.760194 (+ 120us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:48:14.760704 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.760722 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.762578 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.762583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.764660 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:14.764671 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.766645 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.766651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:14.766816 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.766848 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 02:48:14.767752 (+ 904us) client_negotiation.cc:240] Negotiation successful
0929 02:48:14.767891 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18081,"mutex_wait_us":3,"thread_start_us":14828,"threads_started":1}
I20260929 02:48:14.769011 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:14.741143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.0.97.2:36399)
0929 02:48:14.750150 (+ 9007us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:14.750161 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:14.758802 (+ 8641us) server_negotiation.cc:407] Connection header received
0929 02:48:14.758950 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:14.759420 (+ 470us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:14.759653 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:14.759842 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:48:14.760914 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.762466 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.764801 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:14.766188 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:14.766315 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:14.766502 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:14.768484 (+ 1982us) server_negotiation.cc:299] Negotiation successful
0929 02:48:14.768678 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8850,"thread_start_us":8705,"threads_started":1}
I20260929 02:48:14.770005 12128 tablet_service.cc:1467] Processing CreateTablet for tablet 474c90025c2a474c8203002514e4b49d (DEFAULT_TABLE table=test-table [id=1d7c3f1fe444464d966450c14b96d39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:48:14.770141 11851 tablet_service.cc:1467] Processing CreateTablet for tablet 474c90025c2a474c8203002514e4b49d (DEFAULT_TABLE table=test-table [id=1d7c3f1fe444464d966450c14b96d39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:48:14.771368 12128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474c90025c2a474c8203002514e4b49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:48:14.771504 11851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474c90025c2a474c8203002514e4b49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:48:14.772440 11994 tablet_service.cc:1467] Processing CreateTablet for tablet 474c90025c2a474c8203002514e4b49d (DEFAULT_TABLE table=test-table [id=1d7c3f1fe444464d966450c14b96d39b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:48:14.773741 11994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 474c90025c2a474c8203002514e4b49d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:48:14.784109 12212 tablet_bootstrap.cc:492] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Bootstrap starting.
I20260929 02:48:14.787235 12213 tablet_bootstrap.cc:492] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Bootstrap starting.
I20260929 02:48:14.788578 12212 tablet_bootstrap.cc:654] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Neither blocks nor log segments found. Creating new log.
I20260929 02:48:14.788793 12214 tablet_bootstrap.cc:492] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Bootstrap starting.
I20260929 02:48:14.790357 12212 log.cc:824] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:48:14.791406 12213 tablet_bootstrap.cc:654] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Neither blocks nor log segments found. Creating new log.
I20260929 02:48:14.791747 12214 tablet_bootstrap.cc:654] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Neither blocks nor log segments found. Creating new log.
I20260929 02:48:14.792730 12214 log.cc:824] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Log is configured to *not* fsync() on all Append() calls
I20260929 02:48:14.793017 12213 log.cc:824] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:48:14.795291 12214 tablet_bootstrap.cc:492] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: No bootstrap required, opened a new log
I20260929 02:48:14.795552 12214 ts_tablet_manager.cc:1397] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 02:48:14.796038 12212 tablet_bootstrap.cc:492] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: No bootstrap required, opened a new log
I20260929 02:48:14.796308 12212 ts_tablet_manager.cc:1397] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260929 02:48:14.797676 12213 tablet_bootstrap.cc:492] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: No bootstrap required, opened a new log
I20260929 02:48:14.797952 12213 ts_tablet_manager.cc:1397] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 02:48:14.803898 12214 raft_consensus.cc:348] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:14.804304 12214 raft_consensus.cc:374] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:48:14.804435 12214 raft_consensus.cc:729] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9ce1467203a4bb8b9720d350cda14ff, State: Initialized, Role: FOLLOWER
I20260929 02:48:14.805130 12214 consensus_queue.cc:260] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:14.806656 12059 heartbeater.cc:503] Master 127.0.97.62:34269 was elected leader, sending a full tablet report...
I20260929 02:48:14.807436 12212 raft_consensus.cc:348] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:14.807926 12212 raft_consensus.cc:374] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:48:14.808060 12212 raft_consensus.cc:729] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91e3154d95f741e5958a38f87afcae2e, State: Initialized, Role: FOLLOWER
I20260929 02:48:14.808535 12214 ts_tablet_manager.cc:1428] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
I20260929 02:48:14.808809 12212 consensus_queue.cc:260] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:14.808948 12213 raft_consensus.cc:348] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:14.809449 12213 raft_consensus.cc:374] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:48:14.809607 12213 raft_consensus.cc:729] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b85fa482037452ba59e4cbd7654836f, State: Initialized, Role: FOLLOWER
I20260929 02:48:14.809927 12193 heartbeater.cc:503] Master 127.0.97.62:34269 was elected leader, sending a full tablet report...
I20260929 02:48:14.810392 12213 consensus_queue.cc:260] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:14.811811 12212 ts_tablet_manager.cc:1428] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260929 02:48:14.813467 12213 ts_tablet_manager.cc:1428] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
W20260929 02:48:14.853657 12060 tablet.cc:2369] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:48:14.857671 11917 tablet.cc:2369] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:48:14.914484 12194 tablet.cc:2369] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:48:15.051057 12218 raft_consensus.cc:482] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:48:15.051394 12218 raft_consensus.cc:504] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:15.053042 12218 leader_election.cc:288] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91e3154d95f741e5958a38f87afcae2e (127.0.97.3:38407), 2b85fa482037452ba59e4cbd7654836f (127.0.97.1:44883)
I20260929 02:48:15.062175 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.053781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:32919 (local address 127.0.97.3:38407)
0929 02:48:15.053957 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:15.053961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:15.054497 (+ 536us) server_negotiation.cc:407] Connection header received
0929 02:48:15.054740 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:15.054745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:15.054804 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:15.054898 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:48:15.055515 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.056700 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.059646 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.060977 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.061095 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.061270 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:15.061782 (+ 512us) server_negotiation.cc:299] Negotiation successful
0929 02:48:15.061935 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:48:15.062176 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.053551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38953 (local address 127.0.97.1:44883)
0929 02:48:15.053771 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:15.053778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:15.054903 (+ 1125us) server_negotiation.cc:407] Connection header received
0929 02:48:15.055250 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:15.055254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:15.055314 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:15.055423 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:48:15.055984 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.056990 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.059646 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.060965 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.061097 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.061215 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:15.061783 (+ 568us) server_negotiation.cc:299] Negotiation successful
0929 02:48:15.061914 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:48:15.062428 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.053926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44883 (local address 127.0.97.2:38953)
0929 02:48:15.054788 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:15.054806 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:15.055026 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:15.055437 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:15.055444 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:15.055474 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:48:15.055801 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:15.055811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.057113 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.057119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:15.059450 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:15.059469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.061276 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.061280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:15.061548 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.061645 (+ 97us) client_negotiation.cc:769] Sending connection context
0929 02:48:15.061741 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 02:48:15.061941 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":213,"thread_start_us":541,"threads_started":1}
I20260929 02:48:15.062484 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.053755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38407 (local address 127.0.97.2:32919)
0929 02:48:15.054372 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0929 02:48:15.054390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:48:15.054589 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:48:15.054929 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:48:15.054935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:48:15.054958 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:48:15.055343 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:15.055355 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.056831 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.056837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:15.059453 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:48:15.059469 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.061273 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.061280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:48:15.061518 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.061642 (+ 124us) client_negotiation.cc:769] Sending connection context
0929 02:48:15.061741 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:48:15.061941 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":203,"thread_start_us":334,"threads_started":1}
I20260929 02:48:15.063544 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c90025c2a474c8203002514e4b49d" candidate_uuid: "a9ce1467203a4bb8b9720d350cda14ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e3154d95f741e5958a38f87afcae2e" is_pre_election: true
I20260929 02:48:15.063553 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c90025c2a474c8203002514e4b49d" candidate_uuid: "a9ce1467203a4bb8b9720d350cda14ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b85fa482037452ba59e4cbd7654836f" is_pre_election: true
I20260929 02:48:15.064036 12148 raft_consensus.cc:2454] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ce1467203a4bb8b9720d350cda14ff in term 0.
I20260929 02:48:15.064037 11871 raft_consensus.cc:2454] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9ce1467203a4bb8b9720d350cda14ff in term 0.
I20260929 02:48:15.065099 11948 leader_election.cc:302] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e3154d95f741e5958a38f87afcae2e, a9ce1467203a4bb8b9720d350cda14ff; no voters:
I20260929 02:48:15.065554 12218 raft_consensus.cc:2781] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:48:15.065718 12218 raft_consensus.cc:482] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:48:15.065827 12218 raft_consensus.cc:3037] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:48:15.068004 12218 raft_consensus.cc:504] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:15.068702 12218 leader_election.cc:288] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [CANDIDATE]: Term 1 election: Requested vote from peers 91e3154d95f741e5958a38f87afcae2e (127.0.97.3:38407), 2b85fa482037452ba59e4cbd7654836f (127.0.97.1:44883)
I20260929 02:48:15.069142 12148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c90025c2a474c8203002514e4b49d" candidate_uuid: "a9ce1467203a4bb8b9720d350cda14ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91e3154d95f741e5958a38f87afcae2e"
I20260929 02:48:15.069171 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "474c90025c2a474c8203002514e4b49d" candidate_uuid: "a9ce1467203a4bb8b9720d350cda14ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b85fa482037452ba59e4cbd7654836f"
I20260929 02:48:15.069352 12148 raft_consensus.cc:3037] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:48:15.069387 11871 raft_consensus.cc:3037] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:48:15.071467 12148 raft_consensus.cc:2454] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ce1467203a4bb8b9720d350cda14ff in term 1.
I20260929 02:48:15.071717 11871 raft_consensus.cc:2454] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9ce1467203a4bb8b9720d350cda14ff in term 1.
I20260929 02:48:15.071943 11948 leader_election.cc:302] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91e3154d95f741e5958a38f87afcae2e, a9ce1467203a4bb8b9720d350cda14ff; no voters:
I20260929 02:48:15.072257 12218 raft_consensus.cc:2781] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:48:15.072750 12218 raft_consensus.cc:686] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 1 LEADER]: Becoming Leader. State: Replica: a9ce1467203a4bb8b9720d350cda14ff, State: Running, Role: LEADER
I20260929 02:48:15.073217 12218 consensus_queue.cc:237] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } }
I20260929 02:48:15.077936 11702 catalog_manager.cc:5495] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff reported cstate change: term changed from 0 to 1, leader changed from <none> to a9ce1467203a4bb8b9720d350cda14ff (127.0.97.2). New cstate: current_term: 1 leader_uuid: "a9ce1467203a4bb8b9720d350cda14ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9ce1467203a4bb8b9720d350cda14ff" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36399 } health_report { overall_health: HEALTHY } } }
I20260929 02:48:15.139804 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.134484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42726 (local address 127.0.97.2:36399)
0929 02:48:15.134695 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:15.134702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:15.134725 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:48:15.134835 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:15.134839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:15.134934 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:15.135092 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:48:15.135456 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.136150 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.137614 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.138081 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.138205 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.138677 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:48:15.139230 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:48:15.139393 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:15.139448 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 02:48:15.139521 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:48:15.148084 12148 raft_consensus.cc:1261] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 1 FOLLOWER]: Refusing update from remote peer a9ce1467203a4bb8b9720d350cda14ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:48:15.148065 11871 raft_consensus.cc:1261] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 1 FOLLOWER]: Refusing update from remote peer a9ce1467203a4bb8b9720d350cda14ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:48:15.149164 12218 consensus_queue.cc:1035] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b85fa482037452ba59e4cbd7654836f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:48:15.149542 12223 consensus_queue.cc:1035] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "91e3154d95f741e5958a38f87afcae2e" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:48:15.161233 12226 mvcc.cc:205] Tried to move back new op lower bound from 7334502789715787776 to 7334502789423824896. Current Snapshot: MvccSnapshot[applied={T|T < 7334502789715787776}]
I20260929 02:48:15.162417 12228 mvcc.cc:205] Tried to move back new op lower bound from 7334502789715787776 to 7334502789423824896. Current Snapshot: MvccSnapshot[applied={T|T < 7334502789715787776}]
I20260929 02:48:15.176576 11702 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:48:15.176891 11702 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41962:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:48:15.180709 11702 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29c52a94a1064f00947f9e94710ac433: Sending DeleteTablet for 3 replicas of tablet 474c90025c2a474c8203002514e4b49d
I20260929 02:48:15.181849 12128 tablet_service.cc:1514] Processing DeleteTablet for tablet 474c90025c2a474c8203002514e4b49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:48:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44748
I20260929 02:48:15.181911 11994 tablet_service.cc:1514] Processing DeleteTablet for tablet 474c90025c2a474c8203002514e4b49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:48:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42712
I20260929 02:48:15.181980 11851 tablet_service.cc:1514] Processing DeleteTablet for tablet 474c90025c2a474c8203002514e4b49d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:48:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59452
I20260929 02:48:15.185046 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 11783
I20260929 02:48:15.191911 12238 tablet_replica.cc:330] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: stopping tablet replica
I20260929 02:48:15.192459 12238 raft_consensus.cc:2229] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:48:15.193100 12238 raft_consensus.cc:2258] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:48:15.194191 12237 tablet_replica.cc:330] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: stopping tablet replica
I20260929 02:48:15.194607 12237 raft_consensus.cc:2229] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:48:15.195107 12237 raft_consensus.cc:2258] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:48:15.195468 12236 tablet_replica.cc:330] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: stopping tablet replica
I20260929 02:48:15.195914 12238 ts_tablet_manager.cc:1905] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:48:15.197177 12237 ts_tablet_manager.cc:1905] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:48:15.200924 12236 raft_consensus.cc:2229] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:48:15.201936 12236 raft_consensus.cc:2258] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:48:15.203977 12237 ts_tablet_manager.cc:1918] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:48:15.204176 12237 log.cc:1196] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/474c90025c2a474c8203002514e4b49d
I20260929 02:48:15.204749 12237 ts_tablet_manager.cc:1939] T 474c90025c2a474c8203002514e4b49d P 91e3154d95f741e5958a38f87afcae2e: Deleting consensus metadata
I20260929 02:48:15.204952 12238 ts_tablet_manager.cc:1918] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 29 02:48:15 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:48:15.205243 12238 log.cc:1196] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/474c90025c2a474c8203002514e4b49d
I20260929 02:48:15.205987 12238 ts_tablet_manager.cc:1939] T 474c90025c2a474c8203002514e4b49d P 2b85fa482037452ba59e4cbd7654836f: Deleting consensus metadata
I20260929 02:48:15.206709 11688 catalog_manager.cc:4841] TS 91e3154d95f741e5958a38f87afcae2e (127.0.97.3:38407): tablet 474c90025c2a474c8203002514e4b49d (table test-table [id=1d7c3f1fe444464d966450c14b96d39b]) successfully deleted
I20260929 02:48:15.207959 11690 catalog_manager.cc:4841] TS 2b85fa482037452ba59e4cbd7654836f (127.0.97.1:44883): tablet 474c90025c2a474c8203002514e4b49d (table test-table [id=1d7c3f1fe444464d966450c14b96d39b]) successfully deleted
I20260929 02:48:15.209146 12236 ts_tablet_manager.cc:1905] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:48:15.212994 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.186771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.0.97.1:44883)
0929 02:48:15.186943 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:15.186949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:15.194629 (+ 7680us) server_negotiation.cc:407] Connection header received
0929 02:48:15.194909 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:15.194915 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:15.195016 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:15.195094 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:15.196909 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.198038 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.200059 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.200735 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.200839 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.206701 (+ 5862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:15.206713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:15.206740 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:15.206785 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:15.210091 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:15.210800 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:15.210806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:15.210809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:15.210930 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:15.211336 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:15.211340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:15.211342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:15.211712 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:15.212015 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:15.212368 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 02:48:15.212614 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:48:15.213734 11911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:48:15.218206 12236 ts_tablet_manager.cc:1918] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:48:15.218408 12236 log.cc:1196] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/474c90025c2a474c8203002514e4b49d
I20260929 02:48:15.219020 12236 ts_tablet_manager.cc:1939] T 474c90025c2a474c8203002514e4b49d P a9ce1467203a4bb8b9720d350cda14ff: Deleting consensus metadata
I20260929 02:48:15.220932 11689 catalog_manager.cc:4841] TS a9ce1467203a4bb8b9720d350cda14ff (127.0.97.2:36399): tablet 474c90025c2a474c8203002514e4b49d (table test-table [id=1d7c3f1fe444464d966450c14b96d39b]) successfully deleted
I20260929 02:48:15.306536 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 11783
I20260929 02:48:15.329664 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 11922
Sep 29 02:48:15 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:48:15.347852 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.330429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.0.97.2:36399)
0929 02:48:15.330603 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:15.330607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:15.330666 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:48:15.330908 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:15.330912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:15.330978 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:15.331061 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:15.332730 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.333524 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.335425 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.335878 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.335972 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.341209 (+ 5237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:15.341227 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:15.341245 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:15.341289 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:15.344475 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:15.345558 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:15.345567 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:15.345569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:15.345718 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:15.346121 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:15.346124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:15.346126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:15.346456 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:15.346702 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:15.347109 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 02:48:15.347276 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:48:15.348582 12054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:48:15.464099 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 11922
I20260929 02:48:15.485016 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 12065
Sep 29 02:48:15 dist-test-slave-6tjh krb5kdc[11227](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:48:15.499821 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:48:15.485789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.0.97.3:38407)
0929 02:48:15.485965 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 02:48:15.485972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:48:15.485990 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:48:15.486153 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:48:15.486157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:48:15.486221 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:48:15.486312 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:48:15.487548 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.488303 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.489595 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:48:15.489979 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:48:15.490071 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:48:15.494210 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:48:15.494225 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:48:15.494241 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:48:15.494274 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:48:15.497157 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:15.497819 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:15.497826 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:15.497828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:15.497943 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:48:15.498267 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:48:15.498270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:48:15.498272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:48:15.498768 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:48:15.498981 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 02:48:15.499328 (+ 347us) server_negotiation.cc:299] Negotiation successful
0929 02:48:15.499492 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:48:15.500432 12188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:48:15.592046 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 12065
I20260929 02:48:15.608027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 11671
I20260929 02:48:15.609202 11733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:48:15.721913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 11671
I20260929 02:48:15.745916 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:48:16.091941 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:48:16.092154 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:48:16.438114 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:48:16.438 UTC [11257] LOG: received smart shutdown request
2026-09-29 02:48:16.442 UTC [11257] LOG: background worker "logical replication launcher" (PID 11266) exited with exit code 1
2026-09-29 02:48:16.442 UTC [11261] LOG: shutting down
2026-09-29 02:48:16.626 UTC [11257] LOG: database system is shut down
2026-09-29T02:48:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50003 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:48:16 dist-test-slave-6tjh krb5kdc[12252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:48:16 dist-test-slave-6tjh krb5kdc[12252](info): set up 2 sockets
Sep 29 02:48:16 dist-test-slave-6tjh krb5kdc[12252](info): commencing operation
krb5kdc: starting...
W20260929 02:48:17.857010 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:48:18 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:48:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:48:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260929 02:48:18.184525 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskMdzdaQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/postgres -l logfile start
2026-09-29 02:48:20.768 UTC [12282] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 02:48:20.768 UTC [12282] LOG: listening on IPv4 address "127.0.97.20", port 48009
2026-09-29 02:48:20.772 UTC [12282] LOG: listening on Unix socket "/tmp/.s.PGSQL.48009"
2026-09-29 02:48:20.794 UTC [12285] LOG: database system was shut down at 2026-09-29 02:48:19 UTC
2026-09-29 02:48:20.797 UTC [12282] LOG: database system is ready to accept connections
I20260929 02:48:21.800505 388 mini_postgres.cc:96] Postgres bound to 48009
2026-09-29 02:48:21.817 UTC [12293] FATAL: database "slave" does not exist
127.0.97.20:48009 - accepting connections
I20260929 02:48:21.818061 388 mini_ranger.cc:161] Starting Ranger...
I20260929 02:48:21.844731 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 02:48:22.358592 388 mini_ranger.cc:87] Created ranger Postgres database
I20260929 02:48:22.358711 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-admin
2026-09-29 02:48:22,652 [I] DB FLAVOR :POSTGRES
2026-09-29 02:48:22,652 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:48:22,652 [I] Checking connection..
2026-09-29 02:48:22,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 02:48:22,915 [I] Checking connection passed.
2026-09-29 02:48:22,915 [I] --------- Verifying version history table ---------
2026-09-29 02:48:22,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:48:23,156 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:48:23,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:48:23,423 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 02:48:23,423 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 02:48:23,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 02:48:23.674 UTC [12357] WARNING: there is no transaction in progress
2026-09-29 02:48:23,681 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 02:48:23,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 02:48:23,958 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 02:48:23,958 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:48:23,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 02:48:24,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 02:48:24,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6tjh.c.gcp-upstream.internal','N') ;"
2026-09-29 02:48:24,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:48:24,978 [I] Table x_portal_user does not exist in database ranger
2026-09-29 02:48:24,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:48:25,237 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 02:48:25,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:48:25,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:48:25,774 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 02:48:25,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 02:48:26.792 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.809 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.826 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.844 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.861 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.879 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.936 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.943 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.958 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.968 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.978 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:26.994 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:27.004 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:27.012 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:27.020 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:27.028 UTC [12465] WARNING: there is no transaction in progress
2026-09-29 02:48:27,738 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 02:48:27,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 02:48:27,965 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 02:48:27,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 02:48:28,229 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 02:48:28,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 02:48:28,489 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 02:48:28,489 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 02:48:28,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 02:48:29,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6tjh.c.gcp-upstream.internal';"
2026-09-29 02:48:29,277 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 02:48:29,278 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 02:48:29,510 [I] DB_PATCHES have already been applied
I20260929 02:48:29.516222 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskMdzdaQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 02:48:29.516352 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260929 02:48:29.531345 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:47541
Sep 29, 2026 2:48:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:48:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:48:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 2:48:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 2:48:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 2:48:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 2:48:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 02:48:30.679019 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.861s user 0.002s sys 0.037s
Sep 29, 2026 2:48:51 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 2:48:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 2:48:51 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 2:48:51 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 2:48:52 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 02:48:53.482722 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 02:48:53 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650133, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 02:48:53.653935 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 02:48:53.675633 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 02:48:54.678639 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 02:48:54.678759 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms
I20260929 02:48:54.678812 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:48009
2026-09-29 02:48:55,033 [I] DB FLAVOR :POSTGRES
2026-09-29 02:48:55,033 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 02:48:55,033 [I] Checking connection
2026-09-29 02:48:55,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 02:48:55,269 [I] connection success
2026-09-29 02:48:55,269 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 02:48:55,269 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 02:48:55,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 02:48:55,514 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 02:48:55,515 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 02:48:55,515 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 02:48:55,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:48009/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 02:48:55,778 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 02:48:56.282843 388 mini_ranger_kms.cc:310] Created kms service
I20260929 02:48:56.510545 388 mini_ranger_kms.cc:326] Created kudu user
I20260929 02:48:56.558797 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 02:48:56 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650133, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:48:57.022655 388 mini_ranger_kms.cc:384] Added ranger policy
I20260929 02:48:57.023079 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskMdzdaQ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskMdzdaQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 02:48:57.023149 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260929 02:48:57.038906 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12646
I20260929 02:48:57.039058 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:48643
Sep 29, 2026 2:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 2:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 2:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 2:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 2:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 2:48:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 02:48:59.383432 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.729s user 0.069s sys 0.299s
I20260929 02:48:59.383695 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 02:48:59.383883 388 mini_ranger_kms.cc:401] 127.0.97.20:48643/kms/v1/keys
Sep 29 02:48:59 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650139, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:48:59 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650139, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 02:49:00 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650139, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 29 02:49:02 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:49:02.513276 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43349
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43349
--ranger_config_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:02.852603 12696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:02.852913 12696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:02.853072 12696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:02.860273 12696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:02.860383 12696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:02.860481 12696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:02.860602 12696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:02.860687 12696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:02.871490 12696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:48643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43349
--ranger_config_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:02.872905 12696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:02.874244 12696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:02.887208 12704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:02.887283 12702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:02.887244 12701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:02.888016 12696 server_base.cc:1034] running on GCE node
I20260929 02:49:02.888934 12696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:02.890506 12696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:02.891717 12696 hybrid_clock.cc:648] HybridClock initialized: now 1790650142891679 us; error 37 us; skew 500 ppm
I20260929 02:49:02.892081 12696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:02 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:02.900517 12696 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:02.902200 12696 webserver.cc:460] Webserver started at http://127.0.97.62:38057/ using document root <none> and password file <none>
I20260929 02:49:02.902915 12696 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:02.903051 12696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:02.903369 12696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:49:02 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:02.957927 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "53a6221510164129a35b2a3246496c66"
format_stamp: "Formatted at 2026-09-29 02:49:02 on dist-test-slave-6tjh"
server_key: "ddbbf41b3ce5c54c956349ec6f0e6139"
server_key_iv: "bc6fd64d84bb63fa083256a78281e654"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:02.958863 12696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "53a6221510164129a35b2a3246496c66"
format_stamp: "Formatted at 2026-09-29 02:49:02 on dist-test-slave-6tjh"
server_key: "ddbbf41b3ce5c54c956349ec6f0e6139"
server_key_iv: "bc6fd64d84bb63fa083256a78281e654"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:02.964927 12696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 02:49:02.983238 12711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:02.985095 12696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:49:02.985299 12696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "53a6221510164129a35b2a3246496c66"
format_stamp: "Formatted at 2026-09-29 02:49:02 on dist-test-slave-6tjh"
server_key: "ddbbf41b3ce5c54c956349ec6f0e6139"
server_key_iv: "bc6fd64d84bb63fa083256a78281e654"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:02.985566 12696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:03.006475 12696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:03.010983 12696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:03.011350 12696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:03.032577 12696 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43349
I20260929 02:49:03.032581 12762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43349 every 8 connection(s)
I20260929 02:49:03.034113 12696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:03.038503 12763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:03.038609 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 12696
I20260929 02:49:03.038830 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:49:03.039726 388 external_mini_cluster.cc:1428] Setting key f791de3116cfef66bf4963c645244b13
I20260929 02:49:03.054374 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Bootstrap starting.
Sep 29 02:49:03 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:03.057843 12763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:03.059302 12763 log.cc:824] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:03.062417 12763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: No bootstrap required, opened a new log
I20260929 02:49:03.064965 12766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:03.042425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42940 (local address 127.0.97.62:43349)
0929 02:49:03.043308 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:03.043335 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:03.043389 (+ 54us) server_negotiation.cc:407] Connection header received
0929 02:49:03.046089 (+ 2700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:03.046120 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:03.046710 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:03.047203 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:03.048365 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.050629 (+ 2264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.051891 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:03.053205 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:03.053399 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:03.057278 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:03.057288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:03.057307 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:03.057366 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:03.060994 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:03.061651 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:03.061664 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:03.061670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:03.061826 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:03.062208 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:03.062214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:03.062217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:03.062771 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:03.063071 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:03.063513 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 02:49:03.063907 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":418,"threads_started":1}
I20260929 02:49:03.071453 12763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:03.071889 12763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:03.072037 12763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a6221510164129a35b2a3246496c66, State: Initialized, Role: FOLLOWER
I20260929 02:49:03.072772 12763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:03.073012 12763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:03.073144 12763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:03.073330 12763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:03.075537 12763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:03.076047 12763 leader_election.cc:302] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53a6221510164129a35b2a3246496c66; no voters:
I20260929 02:49:03.076736 12763 leader_election.cc:288] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:03.076843 12768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:03.078702 12763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:03.080286 12768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 LEADER]: Becoming Leader. State: Replica: 53a6221510164129a35b2a3246496c66, State: Running, Role: LEADER
I20260929 02:49:03.080987 12768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:03.084573 12769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53a6221510164129a35b2a3246496c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } } }
I20260929 02:49:03.085052 12769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:03.084561 12770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a6221510164129a35b2a3246496c66. Latest consensus state: current_term: 1 leader_uuid: "53a6221510164129a35b2a3246496c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } } }
I20260929 02:49:03.085332 12770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:03.086911 12780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:03.087709 12763 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:49:03.099953 12780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:03.113912 12780 catalog_manager.cc:1347] Generated new cluster ID: df076cffe8954e279d539c8e504f7f61
I20260929 02:49:03.114148 12780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:03.131390 12780 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:03.133013 12780 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:03.139590 12780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Generated new TSK 0
I20260929 02:49:03.140933 12780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:49:03.986763 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43349
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:49:04 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 02:49:04.489988 12808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:04.490417 12808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:04.490631 12808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:04.502607 12808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:04.502781 12808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:04.503029 12808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:49:04.519309 12808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:48643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:04.521320 12808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:04.523133 12808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:04.541385 12818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:04.542090 12819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:04.554706 12821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:04.555044 12808 server_base.cc:1034] running on GCE node
I20260929 02:49:04.555848 12808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:04.557437 12808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:04.558841 12808 hybrid_clock.cc:648] HybridClock initialized: now 1790650144558770 us; error 73 us; skew 500 ppm
I20260929 02:49:04.559319 12808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:04 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:04.563719 12808 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:49:04.567327 12808 webserver.cc:460] Webserver started at http://127.0.97.1:35115/ using document root <none> and password file <none>
I20260929 02:49:04.568188 12808 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:04.568365 12808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:04.568771 12808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:49:04 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:04.588294 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "ad107929b651405989339fb5edd15313"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-6tjh"
server_key: "9e777a7c1f3017ef16127b4b03b7606c"
server_key_iv: "2c498447a55d94e63d655e58d61771e8"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:04.589665 12808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ad107929b651405989339fb5edd15313"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-6tjh"
server_key: "9e777a7c1f3017ef16127b4b03b7606c"
server_key_iv: "2c498447a55d94e63d655e58d61771e8"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:04.627143 12808 fs_manager.cc:691] Time spent creating directory manager: real 0.037s user 0.017s sys 0.016s
I20260929 02:49:04.667253 12828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:04.669584 12808 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.000s sys 0.002s
I20260929 02:49:04.669970 12808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "ad107929b651405989339fb5edd15313"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-6tjh"
server_key: "9e777a7c1f3017ef16127b4b03b7606c"
server_key_iv: "2c498447a55d94e63d655e58d61771e8"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:04.670356 12808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:04.749665 12808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:04.765658 12808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:04.766362 12808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:04.768683 12808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:04.771054 12808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:04.771245 12808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 02:49:04.771488 12808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:04.771620 12808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 02:49:04 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:04.951936 12808 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43681
I20260929 02:49:04.951938 12940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43681 every 8 connection(s)
I20260929 02:49:04.953504 12808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:04.961334 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 12808
I20260929 02:49:04.961609 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:04.962525 388 external_mini_cluster.cc:1428] Setting key b45d5056351a3dc53c385161299d4a46
Sep 29 02:49:04 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:04.985749 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.957689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43349 (local address 127.0.97.1:35949)
0929 02:49:04.958865 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:04.958962 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:04.961246 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:04.962706 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:04.962722 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:04.963460 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:04.964480 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.964512 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.966199 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.966208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.969098 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:04.969129 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.971173 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.971184 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:04.971620 (+ 436us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.972741 (+ 1121us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:04.972780 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:04.976034 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:04.980369 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:04.980381 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:04.980401 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:04.981083 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:04.981653 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:04.981658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:04.981662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:04.981937 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:04.982785 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:04.982804 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:04.983282 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 02:49:04.983612 (+ 330us) client_negotiation.cc:240] Negotiation successful
0929 02:49:04.984216 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"thread_start_us":505,"threads_started":1}
I20260929 02:49:04.987299 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:04.958335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35949 (local address 127.0.97.62:43349)
0929 02:49:04.958879 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:04.958886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:04.959112 (+ 226us) server_negotiation.cc:407] Connection header received
0929 02:49:04.961854 (+ 2742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:04.961862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:04.961996 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:04.962131 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:04.964751 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.965965 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.969492 (+ 3527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:04.970423 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:04.970594 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:04.976460 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:04.976465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:04.976469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:04.976508 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:04.980145 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.981255 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.981261 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.981265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.981433 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:04.982081 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:04.982086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:04.982090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:04.982473 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:04.982657 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:04.986467 (+ 3810us) server_negotiation.cc:299] Negotiation successful
0929 02:49:04.986820 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260929 02:49:04.990101 12941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43349
I20260929 02:49:04.990550 12941 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:04.991567 12941 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:49:04.994545 12728 ts_manager.cc:194] Registered new tserver with Master: ad107929b651405989339fb5edd15313 (127.0.97.1:43681)
I20260929 02:49:04.996711 12728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:35949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:49:05.037556 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43349
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:49:05 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260929 02:49:05.506382 12947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:05.506709 12947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:05.506863 12947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:05.514283 12947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:05.514456 12947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:05.514639 12947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:49:05.524755 12947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:48643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:05.526163 12947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:05.527592 12947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:05.542392 12961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:05.542557 12962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:05.542928 12964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:05.543286 12947 server_base.cc:1034] running on GCE node
I20260929 02:49:05.544082 12947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:05.545361 12947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:05.546617 12947 hybrid_clock.cc:648] HybridClock initialized: now 1790650145546574 us; error 36 us; skew 500 ppm
I20260929 02:49:05.547005 12947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:05 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:05.551159 12947 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:49:05.552987 12947 webserver.cc:460] Webserver started at http://127.0.97.2:34267/ using document root <none> and password file <none>
I20260929 02:49:05.553758 12947 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:05.553922 12947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:05.554333 12947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:49:05 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:05.572379 12947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "5a792c0475604dcfb68cd906c73738f2"
format_stamp: "Formatted at 2026-09-29 02:49:05 on dist-test-slave-6tjh"
server_key: "db57abae940406203f6ced8069f96731"
server_key_iv: "36ef9db8b7ed4a961732c0a8a8decc17"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:05.573587 12947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5a792c0475604dcfb68cd906c73738f2"
format_stamp: "Formatted at 2026-09-29 02:49:05 on dist-test-slave-6tjh"
server_key: "db57abae940406203f6ced8069f96731"
server_key_iv: "36ef9db8b7ed4a961732c0a8a8decc17"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:05.581615 12947 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20260929 02:49:05.613356 12971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:05.615655 12947 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.006s sys 0.010s
I20260929 02:49:05.615978 12947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "5a792c0475604dcfb68cd906c73738f2"
format_stamp: "Formatted at 2026-09-29 02:49:05 on dist-test-slave-6tjh"
server_key: "db57abae940406203f6ced8069f96731"
server_key_iv: "36ef9db8b7ed4a961732c0a8a8decc17"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:05.616317 12947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:05.650573 12947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:05.655185 12947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:05.655525 12947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:05.656667 12947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:05.658263 12947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:05.658375 12947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:05.658510 12947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:05.658583 12947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:05.705565 12947 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41093
I20260929 02:49:05.705588 13083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41093 every 8 connection(s)
I20260929 02:49:05.706941 12947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:05.709978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 12947
I20260929 02:49:05.710167 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:05.710744 388 external_mini_cluster.cc:1428] Setting key f17d8184be2e2c0a1546c7aa43d34d1b
Sep 29 02:49:05 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:49:05.757565 13086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.712879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45427 (local address 127.0.97.62:43349)
0929 02:49:05.733008 (+ 20129us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:05.733015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:05.735983 (+ 2968us) server_negotiation.cc:407] Connection header received
0929 02:49:05.737058 (+ 1075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:05.737064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:05.737181 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:05.737299 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:05.739735 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.741170 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.743717 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.744444 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.744628 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.748993 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:05.749002 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:05.749007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:05.749067 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:05.752946 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.754137 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.754142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.754146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.754298 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:05.754882 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:05.754886 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:05.754889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:05.755254 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:05.756761 (+ 1507us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:05.756920 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 02:49:05.757166 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19962,"thread_start_us":19812,"threads_started":1}
I20260929 02:49:05.758255 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:05.713389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43349 (local address 127.0.97.2:45427)
0929 02:49:05.733681 (+ 20292us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:05.733753 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:05.736498 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:05.737756 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:05.737769 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:05.738544 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:05.739485 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:05.739515 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.741466 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.741472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:05.743515 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:05.743532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:05.744734 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:05.744742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:05.745074 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:05.745916 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:05.745948 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:05.748675 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:05.753195 (+ 4520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:05.753208 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:05.753229 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:05.753921 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:05.754440 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:05.754445 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:05.754448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:05.754714 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:05.755407 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:05.755416 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:05.755894 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 02:49:05.756201 (+ 307us) client_negotiation.cc:240] Negotiation successful
0929 02:49:05.756797 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19919,"thread_start_us":19665,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260929 02:49:05.760129 13084 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43349
I20260929 02:49:05.760535 13084 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:05.761509 13084 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
I20260929 02:49:05.765511 12728 ts_manager.cc:194] Registered new tserver with Master: 5a792c0475604dcfb68cd906c73738f2 (127.0.97.2:41093)
I20260929 02:49:05.766686 12728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:49:05.796921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43349
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:06.001073 12941 heartbeater.cc:503] Master 127.0.97.62:43349 was elected leader, sending a full tablet report...
W20260929 02:49:06.199054 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:06.199357 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:06.199447 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:06.206646 13090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:06.206770 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:06.206883 13090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:49:06.216928 13090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:48643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:06.218194 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:06.219444 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:06.233356 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:06.233314 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:06.233273 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:06.234174 13090 server_base.cc:1034] running on GCE node
I20260929 02:49:06.234771 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:06.235899 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:06.237141 13090 hybrid_clock.cc:648] HybridClock initialized: now 1790650146237104 us; error 38 us; skew 500 ppm
I20260929 02:49:06.237540 13090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:06 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:06.241170 13090 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:49:06.242985 13090 webserver.cc:460] Webserver started at http://127.0.97.3:38011/ using document root <none> and password file <none>
I20260929 02:49:06.243746 13090 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:06.243922 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:06.244257 13090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 02:49:06 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:06.260735 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "0195766d4d014dacbf3b01f581f3c455"
format_stamp: "Formatted at 2026-09-29 02:49:06 on dist-test-slave-6tjh"
server_key: "7e0b64318ed8b1516dbb168fafdb1a7f"
server_key_iv: "69c6c24aa3bddf5a3e96246e656fa96a"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:06.261725 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0195766d4d014dacbf3b01f581f3c455"
format_stamp: "Formatted at 2026-09-29 02:49:06 on dist-test-slave-6tjh"
server_key: "7e0b64318ed8b1516dbb168fafdb1a7f"
server_key_iv: "69c6c24aa3bddf5a3e96246e656fa96a"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:06.267364 13090 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:49:06.285076 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:06.286823 13090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:49:06.287047 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "0195766d4d014dacbf3b01f581f3c455"
format_stamp: "Formatted at 2026-09-29 02:49:06 on dist-test-slave-6tjh"
server_key: "7e0b64318ed8b1516dbb168fafdb1a7f"
server_key_iv: "69c6c24aa3bddf5a3e96246e656fa96a"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:06.287302 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:06.307063 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:06.311578 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:06.311910 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:06.313014 13090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:06.314484 13090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:06.314585 13090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:06.314736 13090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:06.314815 13090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:06.354348 13090 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35865
I20260929 02:49:06.354442 13217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35865 every 8 connection(s)
I20260929 02:49:06.355732 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:06.362995 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13090
I20260929 02:49:06.363198 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:06.363634 388 external_mini_cluster.cc:1428] Setting key 54214e1ba4f29b7b47913ca585f13055
Sep 29 02:49:06 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:06.381520 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:06.358694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42219 (local address 127.0.97.62:43349)
0929 02:49:06.359222 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:06.359229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:06.360279 (+ 1050us) server_negotiation.cc:407] Connection header received
0929 02:49:06.363064 (+ 2785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:06.363070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:06.363191 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:06.363301 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:06.365520 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.366685 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.369172 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.369680 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.369794 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:06.374261 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:06.374266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:06.374270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:06.374308 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:06.377584 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.378585 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.378591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.378594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.378731 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.379470 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.379475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.379479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.379853 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:06.380036 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:06.380890 (+ 854us) server_negotiation.cc:299] Negotiation successful
0929 02:49:06.381131 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":257,"threads_started":1}
I20260929 02:49:06.382328 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:06.359229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43349 (local address 127.0.97.3:42219)
0929 02:49:06.360068 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:06.360124 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:06.362561 (+ 2437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:06.363757 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:06.363770 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:06.364422 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:06.365302 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:06.365343 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.366857 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.366863 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:06.368999 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:06.369025 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.369909 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.369917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:06.370215 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:06.371071 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:06.371105 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:06.373926 (+ 2821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:06.377761 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:06.377771 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:06.377791 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:06.378456 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:06.378870 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:06.378874 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:06.378879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:06.379212 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:06.380010 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:06.380019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:06.380439 (+ 420us) client_negotiation.cc:769] Sending connection context
0929 02:49:06.380751 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 02:49:06.381152 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":319,"threads_started":1}
I20260929 02:49:06.383826 13218 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43349
I20260929 02:49:06.384176 13218 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:06.385107 13218 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
I20260929 02:49:06.386551 12728 ts_manager.cc:194] Registered new tserver with Master: 0195766d4d014dacbf3b01f581f3c455 (127.0.97.3:35865)
I20260929 02:49:06.387426 12728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42219
I20260929 02:49:06.392011 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:06.393775 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 12808
Sep 29 02:49:06 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:49:06.411026 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:06.394712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.0.97.1:43681)
0929 02:49:06.395350 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:06.395358 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:06.395387 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:49:06.395495 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:06.395502 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:06.395715 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:06.395867 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:06.397491 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.398624 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.400091 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.400737 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.400849 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:06.405320 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:06.405333 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:06.405348 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:06.405386 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:06.408033 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.408810 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.408818 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.408820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.408965 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.409427 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.409431 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.409433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.409786 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:06.410035 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:06.410399 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 02:49:06.410610 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":319,"threads_started":1}
I20260929 02:49:06.412371 12936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:06.501294 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 12808
I20260929 02:49:06.516929 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 12947
Sep 29 02:49:06 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:49:06.534977 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:06.517855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.0.97.2:41093)
0929 02:49:06.518451 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:06.518462 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:06.518489 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:49:06.518549 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:06.518556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:06.518772 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:06.518913 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:06.520337 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.521634 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.523171 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.523800 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.523932 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:06.528994 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:06.529009 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:06.529024 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:06.529072 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:06.532051 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.532851 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.532859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.532861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.533003 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.533471 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.533475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.533477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.533887 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:06.534135 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:06.534442 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 02:49:06.534659 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":320,"threads_started":1}
I20260929 02:49:06.536315 13079 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:06.619690 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 12947
I20260929 02:49:06.636431 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13090
Sep 29 02:49:06 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:49:06.654395 13227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:06.637634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.97.3:35865)
0929 02:49:06.638355 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:06.638362 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:06.638392 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:49:06.638453 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:06.638460 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:06.638708 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:06.638843 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:06.640240 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.641429 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.642785 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:06.643386 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:06.643489 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:06.648077 (+ 4588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:06.648091 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:06.648106 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:06.648144 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:06.651234 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.652105 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.652113 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.652115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.652252 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:06.652749 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:06.652753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:06.652755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:06.653103 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:06.653461 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:06.653731 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 02:49:06.654010 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":397,"threads_started":1}
I20260929 02:49:06.655893 13213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:06.742664 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13090
I20260929 02:49:06.757916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 12696
I20260929 02:49:06.759310 12758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:06.866288 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 12696
I20260929 02:49:06.888015 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43349
--webserver_interface=127.0.97.62
--webserver_port=38057
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43349
--ranger_config_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:07.228323 13230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:07.228715 13230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:07.228904 13230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:07.236308 13230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:07.236430 13230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:07.236521 13230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:07.236627 13230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:07.236722 13230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:07.247179 13230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43349
--ranger_config_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:07.248620 13230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:07.250044 13230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:07.262794 13238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.262871 13235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.262890 13236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:07.263833 13230 server_base.cc:1034] running on GCE node
I20260929 02:49:07.264725 13230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:07.266266 13230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:07.267521 13230 hybrid_clock.cc:648] HybridClock initialized: now 1790650147267469 us; error 49 us; skew 500 ppm
I20260929 02:49:07.267953 13230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:07 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:07.271324 13230 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:07.273196 13230 webserver.cc:460] Webserver started at http://127.0.97.62:38057/ using document root <none> and password file <none>
I20260929 02:49:07.274016 13230 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:07.274144 13230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:49:07 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:07.329829 13230 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:07.333660 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:07.335248 13230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:07.335418 13230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "53a6221510164129a35b2a3246496c66"
format_stamp: "Formatted at 2026-09-29 02:49:02 on dist-test-slave-6tjh"
server_key: "ddbbf41b3ce5c54c956349ec6f0e6139"
server_key_iv: "bc6fd64d84bb63fa083256a78281e654"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:07.336421 13230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:07.347680 13230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:07.352201 13230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:07.352609 13230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:07.373962 13230 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43349
I20260929 02:49:07.373977 13297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43349 every 8 connection(s)
I20260929 02:49:07.375664 13230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:07.383344 13298 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:49:07.383344 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13230
I20260929 02:49:07.384310 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:43681
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:43349
--webserver_port=35115
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:07.387100 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Bootstrap starting.
I20260929 02:49:07.407974 13298 log.cc:824] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:07.417800 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:49:07.418437 13298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Bootstrap complete.
I20260929 02:49:07.428670 13298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:07.429745 13298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53a6221510164129a35b2a3246496c66, State: Initialized, Role: FOLLOWER
I20260929 02:49:07.430434 13298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:07.430714 13298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:07.430871 13298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:07.431070 13298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:49:07.436177 13298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:07.436877 13298 leader_election.cc:302] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53a6221510164129a35b2a3246496c66; no voters:
I20260929 02:49:07.437584 13298 leader_election.cc:288] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:49:07.437752 13302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:49:07.438932 13302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [term 2 LEADER]: Becoming Leader. State: Replica: 53a6221510164129a35b2a3246496c66, State: Running, Role: LEADER
I20260929 02:49:07.439522 13302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } }
I20260929 02:49:07.440001 13298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:07.447680 13304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53a6221510164129a35b2a3246496c66. Latest consensus state: current_term: 2 leader_uuid: "53a6221510164129a35b2a3246496c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } } }
I20260929 02:49:07.448191 13304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:07.448767 13303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "53a6221510164129a35b2a3246496c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53a6221510164129a35b2a3246496c66" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43349 } } }
I20260929 02:49:07.449080 13303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:07.452769 13312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:07.458577 13312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:07.459754 13298 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 02:49:07.460008 13312 catalog_manager.cc:1259] Loaded cluster ID: df076cffe8954e279d539c8e504f7f61
I20260929 02:49:07.460100 13312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:07.464268 13312 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:07.465780 13312 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Loaded TSK: 0
I20260929 02:49:07.467422 13312 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:49:07.833912 13300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:07.834355 13300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:07.834539 13300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:07.848243 13300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:07.848414 13300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:07.848662 13300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:49:07.864200 13300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:43681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35115
--tserver_master_addrs=127.0.97.62:43349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:07.866080 13300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:07.867937 13300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:07.906098 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.906127 13336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:07.913552 13340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:07.914121 13300 server_base.cc:1034] running on GCE node
I20260929 02:49:07.915119 13300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:07.917128 13300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:07.920497 13300 hybrid_clock.cc:648] HybridClock initialized: now 1790650147920415 us; error 76 us; skew 500 ppm
I20260929 02:49:07.920948 13300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:07 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:07.929873 13300 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:49:07.932399 13300 webserver.cc:460] Webserver started at http://127.0.97.1:35115/ using document root <none> and password file <none>
I20260929 02:49:07.933461 13300 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:07.933660 13300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:49:07 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:07.973824 13300 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.003s sys 0.004s
I20260929 02:49:07.993518 13347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:07.996049 13300 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.009s sys 0.004s
I20260929 02:49:07.996392 13300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "ad107929b651405989339fb5edd15313"
format_stamp: "Formatted at 2026-09-29 02:49:04 on dist-test-slave-6tjh"
server_key: "9e777a7c1f3017ef16127b4b03b7606c"
server_key_iv: "2c498447a55d94e63d655e58d61771e8"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:07.998207 13300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:08.049455 13300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:08.056036 13300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:08.056489 13300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:08.069077 13300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:08.081576 13300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:08.081825 13300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 02:49:08.082038 13300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:08.082176 13300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.167842 13300 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43681
I20260929 02:49:08.167881 13460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43681 every 8 connection(s)
I20260929 02:49:08.170097 13300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:08.172534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13300
I20260929 02:49:08.173386 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41093
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:43349
--webserver_port=34267
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:49:08 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650147, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:08.223012 13463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.174014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43349 (local address 127.0.97.1:53929)
0929 02:49:08.191716 (+ 17702us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.191796 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.194693 (+ 2897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.198106 (+ 3413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.198120 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.198908 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:08.199962 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.199998 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.203090 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.203099 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.205926 (+ 2827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.205952 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.207586 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.207595 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.207946 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.208838 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:08.208871 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:08.212568 (+ 3697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:08.217628 (+ 5060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.217639 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.217659 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.218214 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.218779 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.218785 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.218789 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.219017 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.220102 (+ 1085us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:08.220116 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:08.220462 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.220834 (+ 372us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.221500 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17322,"thread_start_us":14669,"threads_started":1}
I20260929 02:49:08.226588 13464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.174479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53929 (local address 127.0.97.62:43349)
0929 02:49:08.187997 (+ 13518us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.188015 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.192820 (+ 4805us) server_negotiation.cc:407] Connection header received
0929 02:49:08.195863 (+ 3043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.195888 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.196588 (+ 700us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.197154 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:08.200353 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.202810 (+ 2457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.206214 (+ 3404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.207254 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.207568 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.213057 (+ 5489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:08.213072 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:08.213098 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:08.213161 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:08.217371 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.218418 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.218435 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.218441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.218624 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.219159 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.219164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.219168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.219824 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:08.220038 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.224043 (+ 4005us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.225126 (+ 1083us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13219,"thread_start_us":13433,"threads_started":1}
I20260929 02:49:08.230465 13461 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43349
I20260929 02:49:08.230860 13461 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:08.231807 13461 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
W20260929 02:49:08.235244 13461 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:43349 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:49:08.235513 13461 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:08.235832 13461 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
I20260929 02:49:08.237174 13461 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:08.237599 13461 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
W20260929 02:49:08.238816 13461 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:49:08.641021 13465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:08.641383 13465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:08.641571 13465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:08.650058 13465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:08.650259 13465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:08.650470 13465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:49:08.662775 13465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34267
--tserver_master_addrs=127.0.97.62:43349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:08.664268 13465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:08.665729 13465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:08.682173 13477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:08.683902 13465 server_base.cc:1034] running on GCE node
W20260929 02:49:08.684036 13479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:08.684832 13481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:08.685634 13465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:08.686759 13465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:08.688027 13465 hybrid_clock.cc:648] HybridClock initialized: now 1790650148687946 us; error 64 us; skew 500 ppm
I20260929 02:49:08.688376 13465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:08 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:08.692272 13465 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:49:08.697151 13465 webserver.cc:460] Webserver started at http://127.0.97.2:34267/ using document root <none> and password file <none>
I20260929 02:49:08.698220 13465 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:08.698421 13465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:49:08 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:08.736120 13465 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.005s sys 0.008s
I20260929 02:49:08.741564 13490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.743875 13465 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260929 02:49:08.744251 13465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "5a792c0475604dcfb68cd906c73738f2"
format_stamp: "Formatted at 2026-09-29 02:49:05 on dist-test-slave-6tjh"
server_key: "db57abae940406203f6ced8069f96731"
server_key_iv: "36ef9db8b7ed4a961732c0a8a8decc17"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:08.745900 13465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:08.802174 13465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:08.808138 13465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:08.808534 13465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:08.817690 13465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:08.819690 13465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:08.819842 13465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.820067 13465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:08.820170 13465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:08.866439 13465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41093
I20260929 02:49:08.866473 13604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41093 every 8 connection(s)
I20260929 02:49:08.868182 13465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:08.873872 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13465
I20260929 02:49:08.874897 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:48643/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35865
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:43349
--webserver_port=38011
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:49:08 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:49:08 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:08.909940 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.873475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45065 (local address 127.0.97.62:43349)
0929 02:49:08.881129 (+ 7654us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:08.881139 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:08.882381 (+ 1242us) server_negotiation.cc:407] Connection header received
0929 02:49:08.885608 (+ 3227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:08.885618 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:08.885757 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:08.885908 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:08.888538 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.889971 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.893118 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.894002 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.894156 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.899921 (+ 5765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:08.899934 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:08.899941 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:08.900005 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:08.904505 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.906061 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.906071 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.906077 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.906271 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:08.907017 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:08.907024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:08.907027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:08.907543 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:08.907721 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:08.909046 (+ 1325us) server_negotiation.cc:299] Negotiation successful
0929 02:49:08.909399 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7445,"thread_start_us":14239,"threads_started":1}
I20260929 02:49:08.912839 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:08.872499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43349 (local address 127.0.97.2:45065)
0929 02:49:08.881793 (+ 9294us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:08.881851 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:08.884995 (+ 3144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:08.886586 (+ 1591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:08.886604 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:08.887398 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:08.888286 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.888311 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.890246 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.890253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.892845 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:08.892872 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:08.894728 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:08.894740 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:08.895144 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:08.896166 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:08.896208 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:08.899501 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:08.904939 (+ 5438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.904955 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.904979 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.905857 (+ 878us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.906487 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:08.906496 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:08.906503 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:08.906842 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:08.907851 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:08.907864 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:08.908446 (+ 582us) client_negotiation.cc:769] Sending connection context
0929 02:49:08.910447 (+ 2001us) client_negotiation.cc:240] Negotiation successful
0929 02:49:08.911133 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8941,"thread_start_us":8672,"threads_started":1}
I20260929 02:49:08.915107 13605 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43349
I20260929 02:49:08.915483 13605 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:08.916448 13605 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
I20260929 02:49:08.919128 13263 ts_manager.cc:194] Registered new tserver with Master: 5a792c0475604dcfb68cd906c73738f2 (127.0.97.2:41093)
I20260929 02:49:08.921679 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45065
I20260929 02:49:09.238971 13461 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:09.239449 13461 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
I20260929 02:49:09.240751 13263 ts_manager.cc:194] Registered new tserver with Master: ad107929b651405989339fb5edd15313 (127.0.97.1:43681)
I20260929 02:49:09.241983 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53929
W20260929 02:49:09.315686 13609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:09.316233 13609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:09.316443 13609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:09.328529 13609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:09.329149 13609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:09.329380 13609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:49:09.346560 13609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:48643/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38011
--tserver_master_addrs=127.0.97.62:43349
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:09.348713 13609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:09.350634 13609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:09.370646 13615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:09.370668 13616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:09.385934 13619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:09.389062 13609 server_base.cc:1034] running on GCE node
Sep 29 02:49:09 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:09.390223 13609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:09.391636 13609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:09.395133 13609 hybrid_clock.cc:648] HybridClock initialized: now 1790650149394806 us; error 291 us; skew 500 ppm
I20260929 02:49:09.395746 13609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:09 dist-test-slave-6tjh krb5kdc[12252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:09.400764 13609 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:49:09.417676 13609 webserver.cc:460] Webserver started at http://127.0.97.3:38011/ using document root <none> and password file <none>
I20260929 02:49:09.419780 13609 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:09.420352 13609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 02:49:09 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:09.473455 13609 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.005s sys 0.010s
I20260929 02:49:09.489368 13626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:09.494369 13609 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.004s sys 0.011s
I20260929 02:49:09.494827 13609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "0195766d4d014dacbf3b01f581f3c455"
format_stamp: "Formatted at 2026-09-29 02:49:06 on dist-test-slave-6tjh"
server_key: "7e0b64318ed8b1516dbb168fafdb1a7f"
server_key_iv: "69c6c24aa3bddf5a3e96246e656fa96a"
server_key_version: "kuduclusterkey@0"
I20260929 02:49:09.496630 13609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:09.521510 13609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:09.527432 13609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:09.527896 13609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:09.540161 13609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:09.548797 13609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:09.549027 13609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:09.549213 13609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:09.549352 13609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:09.595773 13609 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35865
I20260929 02:49:09.595839 13738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35865 every 8 connection(s)
I20260929 02:49:09.598018 13609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:09.607436 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13609
I20260929 02:49:09.633708 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.609424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.0.97.62:43349)
0929 02:49:09.617111 (+ 7687us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.617119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.617147 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:49:09.617227 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.617233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.617326 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.617485 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.619622 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.620418 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.622424 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.622874 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.623023 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.626551 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.626564 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.626569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.626606 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.629734 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.630552 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.630558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.630562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.630705 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.631116 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.631121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.631126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.631405 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.631701 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.633048 (+ 1347us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.633293 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7500,"thread_start_us":3531,"threads_started":1}
Sep 29 02:49:09 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:09.650035 13742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.601826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43349 (local address 127.0.97.3:38003)
0929 02:49:09.618188 (+ 16362us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.618245 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.620698 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.621944 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.621958 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.622604 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:09.623464 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.623489 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.625149 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.625154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.627192 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.627207 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.628615 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.628623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.628920 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.637861 (+ 8941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:09.637892 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:09.640363 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:09.644607 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:09.644618 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:09.644638 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:09.645385 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:09.645927 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:09.645932 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:09.645936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:09.646247 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:09.647070 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:09.647080 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:09.647571 (+ 491us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.647875 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.648505 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15326,"thread_start_us":15105,"threads_started":1}
I20260929 02:49:09.651543 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.601345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38003 (local address 127.0.97.62:43349)
0929 02:49:09.613263 (+ 11918us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.613270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.618392 (+ 5122us) server_negotiation.cc:407] Connection header received
0929 02:49:09.621115 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.621121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.621206 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.621280 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.623709 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.624936 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.627375 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.628075 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.628169 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.640708 (+ 12539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.640715 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.640718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.640758 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.644388 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.645570 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.645578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.645581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.645747 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.646406 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.646413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.646418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.646831 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.646974 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.650758 (+ 3784us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.651056 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11766,"thread_start_us":15404,"threads_started":1}
I20260929 02:49:09.652969 13739 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43349
I20260929 02:49:09.653340 13739 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:09.654174 13739 heartbeater.cc:511] Master 127.0.97.62:43349 requested a full tablet report, sending...
I20260929 02:49:09.655798 13263 ts_manager.cc:194] Registered new tserver with Master: 0195766d4d014dacbf3b01f581f3c455 (127.0.97.3:35865)
I20260929 02:49:09.656863 13263 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38003
I20260929 02:49:09.663383 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:09.686887 13744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.667073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.0.97.62:43349)
0929 02:49:09.667337 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.667348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.667972 (+ 624us) server_negotiation.cc:407] Connection header received
0929 02:49:09.668216 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.668223 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.668320 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.668450 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:09.670160 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.670967 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.672481 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.673105 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.673271 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.678669 (+ 5398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:09.678678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:09.678684 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:09.678741 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:09.682389 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.683312 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.683319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.683323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.683504 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:09.683981 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:09.683987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:09.683991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:09.684436 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:09.684746 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.686204 (+ 1458us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.686510 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 02:49:09.694173 13263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:09.707937 13263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 29 02:49:09 dist-test-slave-6tjh krb5kdc[12252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260929 02:49:09.781647 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.745510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53894 (local address 127.0.97.1:43681)
0929 02:49:09.766664 (+ 21154us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.766678 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.770881 (+ 4203us) server_negotiation.cc:407] Connection header received
0929 02:49:09.771129 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.771137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.771427 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.771662 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:09.772713 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.774234 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.777174 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.778899 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.779063 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.779354 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.781089 (+ 1735us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.781282 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20935,"thread_start_us":20757,"threads_started":1}
I20260929 02:49:09.787081 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.744356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53894)
0929 02:49:09.770669 (+ 26313us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.770699 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.770918 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.771975 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.771983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.772016 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:09.772451 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.772463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.774409 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.774415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.776959 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.776977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.779535 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.779542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.779768 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.779830 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.779986 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.780206 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26134,"thread_start_us":25872,"threads_started":1}
I20260929 02:49:09.792515 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.743827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35865 (local address 127.0.0.1:55130)
0929 02:49:09.765603 (+ 21776us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.765665 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.765902 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.772504 (+ 6602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.772511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.772535 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:09.772923 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.772936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.781936 (+ 9000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.781943 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.788955 (+ 7012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.788972 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.791570 (+ 2598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.791580 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.791769 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.791816 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.791910 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.792068 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21392,"mutex_wait_us":20,"thread_start_us":32561,"threads_started":1}
I20260929 02:49:09.793545 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.744532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.97.3:35865)
0929 02:49:09.771538 (+ 27006us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.771547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.771583 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:49:09.771681 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.771688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.771971 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.772172 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:09.773789 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.776016 (+ 2227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.789216 (+ 13200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.790942 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.791094 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.791389 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.792976 (+ 1587us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.793185 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26791,"thread_start_us":26633,"threads_started":1}
I20260929 02:49:09.796306 13396 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcd424065704122b059da9108d71227 (DEFAULT_TABLE table=test-table [id=530c80dcecf5451c84597f554c3ec4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:09.801606 13674 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcd424065704122b059da9108d71227 (DEFAULT_TABLE table=test-table [id=530c80dcecf5451c84597f554c3ec4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:09.803637 13674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcd424065704122b059da9108d71227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:09.798166 13396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcd424065704122b059da9108d71227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:09.825281 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.745873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.0.97.2:41093)
0929 02:49:09.767190 (+ 21317us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:09.767203 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:09.784397 (+ 17194us) server_negotiation.cc:407] Connection header received
0929 02:49:09.784902 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:09.784912 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:09.785242 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:09.785468 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:09.787253 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.788965 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.820916 (+ 31951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.822688 (+ 1772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.822883 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.823115 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:09.824707 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0929 02:49:09.824935 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21119,"thread_start_us":20995,"threads_started":1}
I20260929 02:49:09.824481 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:09.744769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41093 (local address 127.0.0.1:53280)
0929 02:49:09.784252 (+ 39483us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:09.784276 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:09.784633 (+ 357us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:09.785808 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:09.785812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:09.785837 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:09.787075 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.787091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.789158 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.789162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.791177 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:09.791193 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:09.823401 (+ 32208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:09.823409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:09.823619 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:09.823657 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:49:09.823793 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:49:09.823978 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31757,"mutex_wait_us":41,"thread_start_us":20321,"threads_started":1}
I20260929 02:49:09.830410 13540 tablet_service.cc:1467] Processing CreateTablet for tablet 8fcd424065704122b059da9108d71227 (DEFAULT_TABLE table=test-table [id=530c80dcecf5451c84597f554c3ec4d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:09.831974 13540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fcd424065704122b059da9108d71227. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:09.854285 13762 tablet_bootstrap.cc:492] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Bootstrap starting.
I20260929 02:49:09.858312 13761 tablet_bootstrap.cc:492] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Bootstrap starting.
I20260929 02:49:09.866144 13762 tablet_bootstrap.cc:654] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.867619 13762 log.cc:824] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:09.868604 13761 tablet_bootstrap.cc:654] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.870118 13761 log.cc:824] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:09.871537 13763 tablet_bootstrap.cc:492] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Bootstrap starting.
I20260929 02:49:09.880743 13762 tablet_bootstrap.cc:492] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: No bootstrap required, opened a new log
I20260929 02:49:09.881075 13762 ts_tablet_manager.cc:1397] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Time spent bootstrapping tablet: real 0.027s user 0.011s sys 0.000s
I20260929 02:49:09.883253 13763 tablet_bootstrap.cc:654] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:09.884788 13763 log.cc:824] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:09.891767 13761 tablet_bootstrap.cc:492] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: No bootstrap required, opened a new log
I20260929 02:49:09.892117 13761 ts_tablet_manager.cc:1397] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.007s
I20260929 02:49:09.893257 13763 tablet_bootstrap.cc:492] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: No bootstrap required, opened a new log
I20260929 02:49:09.893558 13763 ts_tablet_manager.cc:1397] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20260929 02:49:09.905030 13763 raft_consensus.cc:348] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:09.905547 13763 raft_consensus.cc:374] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.905328 13762 raft_consensus.cc:348] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:09.905653 13763 raft_consensus.cc:729] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a792c0475604dcfb68cd906c73738f2, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.905755 13762 raft_consensus.cc:374] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.906543 13763 consensus_queue.cc:260] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:09.906900 13761 raft_consensus.cc:348] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:09.907337 13761 raft_consensus.cc:374] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:09.907438 13761 raft_consensus.cc:729] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad107929b651405989339fb5edd15313, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.905853 13762 raft_consensus.cc:729] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0195766d4d014dacbf3b01f581f3c455, State: Initialized, Role: FOLLOWER
I20260929 02:49:09.908761 13762 consensus_queue.cc:260] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:09.909693 13761 consensus_queue.cc:260] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:09.913690 13739 heartbeater.cc:503] Master 127.0.97.62:43349 was elected leader, sending a full tablet report...
I20260929 02:49:09.913951 13762 ts_tablet_manager.cc:1428] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Time spent starting tablet: real 0.033s user 0.014s sys 0.000s
I20260929 02:49:09.922431 13605 heartbeater.cc:503] Master 127.0.97.62:43349 was elected leader, sending a full tablet report...
I20260929 02:49:09.923620 13763 ts_tablet_manager.cc:1428] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Time spent starting tablet: real 0.030s user 0.012s sys 0.008s
I20260929 02:49:09.924667 13461 heartbeater.cc:503] Master 127.0.97.62:43349 was elected leader, sending a full tablet report...
I20260929 02:49:09.929442 13761 ts_tablet_manager.cc:1428] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Time spent starting tablet: real 0.037s user 0.014s sys 0.010s
W20260929 02:49:10.101553 13740 tablet.cc:2369] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:49:10.130239 13606 tablet.cc:2369] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:10.170440 13767 raft_consensus.cc:482] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:10.170773 13767 raft_consensus.cc:504] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:10.173043 13767 leader_election.cc:288] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0195766d4d014dacbf3b01f581f3c455 (127.0.97.3:35865), ad107929b651405989339fb5edd15313 (127.0.97.1:43681)
W20260929 02:49:10.173197 13462 tablet.cc:2369] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:10.191502 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.173955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:56841 (local address 127.0.97.3:35865)
0929 02:49:10.174222 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:10.174233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:10.184073 (+ 9840us) server_negotiation.cc:407] Connection header received
0929 02:49:10.184390 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:10.184397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:10.184493 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:10.184587 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:10.185510 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.186280 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.189022 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.190340 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.190523 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.190639 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:10.191103 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 02:49:10.191251 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:49:10.191560 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.175146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34549 (local address 127.0.97.1:43681)
0929 02:49:10.175364 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:10.175370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:10.184150 (+ 8780us) server_negotiation.cc:407] Connection header received
0929 02:49:10.184261 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:10.184267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:10.184364 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:10.184482 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:10.185285 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.186307 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.189148 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.190423 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.190553 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.190647 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:10.191182 (+ 535us) server_negotiation.cc:299] Negotiation successful
0929 02:49:10.191318 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:49:10.191586 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.173422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35865 (local address 127.0.97.2:56841)
0929 02:49:10.183854 (+ 10432us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.183889 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:10.184169 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:10.184831 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:10.184836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:10.184862 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:10.185338 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:10.185347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.186450 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.186459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:10.188789 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:10.188813 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.190613 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.190621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:10.190877 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.190914 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 02:49:10.191014 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:49:10.191156 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10219,"mutex_wait_us":131,"spinlock_wait_cycles":30592,"thread_start_us":10791,"threads_started":1}
I20260929 02:49:10.192319 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.173588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.97.2:34549)
0929 02:49:10.183929 (+ 10341us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.183949 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:10.184113 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:10.184564 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:10.184572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:10.184601 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:10.185059 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:10.185074 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.186509 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.186515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:10.188944 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:10.188962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.190832 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.190837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:10.190967 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.191025 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 02:49:10.191841 (+ 816us) client_negotiation.cc:240] Negotiation successful
0929 02:49:10.191988 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10214,"mutex_wait_us":81,"thread_start_us":9790,"threads_started":1}
I20260929 02:49:10.193110 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd424065704122b059da9108d71227" candidate_uuid: "5a792c0475604dcfb68cd906c73738f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0195766d4d014dacbf3b01f581f3c455" is_pre_election: true
I20260929 02:49:10.193456 13416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd424065704122b059da9108d71227" candidate_uuid: "5a792c0475604dcfb68cd906c73738f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad107929b651405989339fb5edd15313" is_pre_election: true
I20260929 02:49:10.193713 13694 raft_consensus.cc:2454] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a792c0475604dcfb68cd906c73738f2 in term 0.
I20260929 02:49:10.194005 13416 raft_consensus.cc:2454] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a792c0475604dcfb68cd906c73738f2 in term 0.
I20260929 02:49:10.194571 13494 leader_election.cc:302] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0195766d4d014dacbf3b01f581f3c455, 5a792c0475604dcfb68cd906c73738f2; no voters:
I20260929 02:49:10.195061 13767 raft_consensus.cc:2781] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:10.195220 13767 raft_consensus.cc:482] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:10.195346 13767 raft_consensus.cc:3037] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:10.197682 13767 raft_consensus.cc:504] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:10.198405 13767 leader_election.cc:288] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [CANDIDATE]: Term 1 election: Requested vote from peers 0195766d4d014dacbf3b01f581f3c455 (127.0.97.3:35865), ad107929b651405989339fb5edd15313 (127.0.97.1:43681)
I20260929 02:49:10.198958 13694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd424065704122b059da9108d71227" candidate_uuid: "5a792c0475604dcfb68cd906c73738f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0195766d4d014dacbf3b01f581f3c455"
I20260929 02:49:10.198956 13416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fcd424065704122b059da9108d71227" candidate_uuid: "5a792c0475604dcfb68cd906c73738f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad107929b651405989339fb5edd15313"
I20260929 02:49:10.199208 13416 raft_consensus.cc:3037] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:10.199199 13694 raft_consensus.cc:3037] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:10.202602 13694 raft_consensus.cc:2454] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a792c0475604dcfb68cd906c73738f2 in term 1.
I20260929 02:49:10.202602 13416 raft_consensus.cc:2454] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a792c0475604dcfb68cd906c73738f2 in term 1.
I20260929 02:49:10.203177 13494 leader_election.cc:302] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0195766d4d014dacbf3b01f581f3c455, 5a792c0475604dcfb68cd906c73738f2; no voters:
I20260929 02:49:10.203694 13767 raft_consensus.cc:2781] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:10.204255 13767 raft_consensus.cc:686] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 1 LEADER]: Becoming Leader. State: Replica: 5a792c0475604dcfb68cd906c73738f2, State: Running, Role: LEADER
I20260929 02:49:10.205284 13767 consensus_queue.cc:237] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } }
I20260929 02:49:10.211126 13262 catalog_manager.cc:5495] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a792c0475604dcfb68cd906c73738f2 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "5a792c0475604dcfb68cd906c73738f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a792c0475604dcfb68cd906c73738f2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:10.333536 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.325439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.0.97.2:41093)
0929 02:49:10.325654 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:10.325659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:10.326339 (+ 680us) server_negotiation.cc:407] Connection header received
0929 02:49:10.326621 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:10.326625 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:10.326729 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:10.326913 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:10.327439 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.328642 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.330387 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.331029 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.331224 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.332086 (+ 862us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:10.332883 (+ 797us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:10.333057 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:10.333156 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 02:49:10.333250 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:49:10.350659 13416 raft_consensus.cc:1261] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 1 FOLLOWER]: Refusing update from remote peer 5a792c0475604dcfb68cd906c73738f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:10.350659 13694 raft_consensus.cc:1261] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 1 FOLLOWER]: Refusing update from remote peer 5a792c0475604dcfb68cd906c73738f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:10.351863 13779 consensus_queue.cc:1035] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0195766d4d014dacbf3b01f581f3c455" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:10.352397 13767 consensus_queue.cc:1035] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad107929b651405989339fb5edd15313" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:10.374272 13783 mvcc.cc:205] Tried to move back new op lower bound from 7334503015823286272 to 7334503015246454784. Current Snapshot: MvccSnapshot[applied={T|T < 7334503015823286272}]
I20260929 02:49:10.378733 13785 mvcc.cc:205] Tried to move back new op lower bound from 7334503015823286272 to 7334503015246454784. Current Snapshot: MvccSnapshot[applied={T|T < 7334503015823286272}]
I20260929 02:49:10.411288 13262 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:10.411602 13262 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:10.416570 13262 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53a6221510164129a35b2a3246496c66: Sending DeleteTablet for 3 replicas of tablet 8fcd424065704122b059da9108d71227
I20260929 02:49:10.417980 13540 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcd424065704122b059da9108d71227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53280
I20260929 02:49:10.418005 13674 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcd424065704122b059da9108d71227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:55130
I20260929 02:49:10.418736 13396 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fcd424065704122b059da9108d71227 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:10 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53894
I20260929 02:49:10.422945 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13300
I20260929 02:49:10.437139 13794 tablet_replica.cc:330] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: stopping tablet replica
I20260929 02:49:10.437695 13794 raft_consensus.cc:2229] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:10.438474 13794 raft_consensus.cc:2258] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:10.440783 13794 ts_tablet_manager.cc:1905] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:10.444322 13795 tablet_replica.cc:330] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: stopping tablet replica
I20260929 02:49:10.445093 13795 raft_consensus.cc:2229] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:10.445770 13795 raft_consensus.cc:2258] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:10.446674 13793 tablet_replica.cc:330] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: stopping tablet replica
I20260929 02:49:10.447302 13793 raft_consensus.cc:2229] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:10.447865 13793 raft_consensus.cc:2258] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:10.448378 13794 ts_tablet_manager.cc:1918] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:10.448711 13794 log.cc:1196] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/8fcd424065704122b059da9108d71227
I20260929 02:49:10.449424 13794 ts_tablet_manager.cc:1939] T 8fcd424065704122b059da9108d71227 P 5a792c0475604dcfb68cd906c73738f2: Deleting consensus metadata
I20260929 02:49:10.450196 13793 ts_tablet_manager.cc:1905] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:10.451393 13250 catalog_manager.cc:4841] TS 5a792c0475604dcfb68cd906c73738f2 (127.0.97.2:41093): tablet 8fcd424065704122b059da9108d71227 (table test-table [id=530c80dcecf5451c84597f554c3ec4d1]) successfully deleted
I20260929 02:49:10.454149 13795 ts_tablet_manager.cc:1905] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:10.459198 13793 ts_tablet_manager.cc:1918] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:10.459573 13793 log.cc:1196] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/8fcd424065704122b059da9108d71227
I20260929 02:49:10.460338 13793 ts_tablet_manager.cc:1939] T 8fcd424065704122b059da9108d71227 P 0195766d4d014dacbf3b01f581f3c455: Deleting consensus metadata
I20260929 02:49:10.462853 13249 catalog_manager.cc:4841] TS 0195766d4d014dacbf3b01f581f3c455 (127.0.97.3:35865): tablet 8fcd424065704122b059da9108d71227 (table test-table [id=530c80dcecf5451c84597f554c3ec4d1]) successfully deleted
I20260929 02:49:10.465384 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.424183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.0.97.1:43681)
0929 02:49:10.424393 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:10.424402 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:10.441369 (+ 16967us) server_negotiation.cc:407] Connection header received
0929 02:49:10.441797 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:10.441808 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:10.441928 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:10.442042 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:10.446664 (+ 4622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.447841 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.451481 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.452254 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.452413 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.456719 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:10.456740 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:10.456761 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:10.456829 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:10.461622 (+ 4793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:10.462732 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:10.462742 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:10.462745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:10.462947 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:10.463461 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:10.463466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:10.463469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:10.464008 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:10.464352 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:10.464706 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 02:49:10.464960 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:49:10.466260 13456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:10.567425 13795 ts_tablet_manager.cc:1918] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:10.567687 13795 log.cc:1196] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/8fcd424065704122b059da9108d71227
I20260929 02:49:10.568459 13795 ts_tablet_manager.cc:1939] T 8fcd424065704122b059da9108d71227 P ad107929b651405989339fb5edd15313: Deleting consensus metadata
I20260929 02:49:10.568471 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13300
I20260929 02:49:10.586799 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13465
W20260929 02:49:10.587298 13247 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:43681 (error 108)
I20260929 02:49:10.610351 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.588841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.0.97.2:41093)
0929 02:49:10.589087 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:10.589095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:10.589121 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:49:10.589203 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:10.589216 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:10.589336 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:10.589466 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:10.592266 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.594044 (+ 1778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.596342 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.597174 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.597329 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.601081 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:10.601101 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:10.601122 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:10.601179 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:10.605914 (+ 4735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:10.606983 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:10.606995 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:10.607002 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:10.607189 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:10.607625 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:10.607630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:10.607633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:10.608089 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:10.608419 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:10.609720 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0929 02:49:10.609961 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:49:10.611111 13600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:10.620388 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.613350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53904)
0929 02:49:10.619954 (+ 6604us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.620136 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6439,"thread_start_us":6281,"threads_started":1}
W20260929 02:49:10.621017 13247 catalog_manager.cc:4584] TS ad107929b651405989339fb5edd15313 (127.0.97.1:43681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8fcd424065704122b059da9108d71227: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
I20260929 02:49:10.675592 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.675097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53908)
0929 02:49:10.675312 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.675429 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:49:10.676890 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.676428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53922)
0929 02:49:10.676643 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.676728 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 02:49:10.737495 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.737000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53926)
0929 02:49:10.737229 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.737326 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:49:10.738631 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.738247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53936)
0929 02:49:10.738407 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.738488 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:49:10.786713 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13465
I20260929 02:49:10.806088 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13609
I20260929 02:49:10.819496 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.818873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53940)
0929 02:49:10.819159 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.819289 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 02:49:10.820739 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.820197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53956)
0929 02:49:10.820424 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:10.820528 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260929 02:49:10.821892 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:10.806950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.97.3:35865)
0929 02:49:10.807414 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:10.807419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:10.807437 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:10.807490 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:10.807493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:10.807565 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:10.807658 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:10.809296 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.810120 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.811542 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:10.811972 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:10.812076 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:10.815113 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:10.815132 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:10.815149 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:10.815200 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:10.818345 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:10.819372 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:10.819382 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:10.819386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:10.819572 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:10.820082 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:10.820089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:10.820092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:10.820629 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:10.820953 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:10.821295 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 02:49:10.821505 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":253,"threads_started":1}
I20260929 02:49:10.822599 13734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:10.913172 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13609
I20260929 02:49:10.930552 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13230
I20260929 02:49:10.932096 13293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:11.047096 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:11.046361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43681 (local address 127.0.0.1:53972)
0929 02:49:11.046672 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:11.046797 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:43681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260929 02:49:11.047284 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13230
I20260929 02:49:11.072744 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 02:49:11.409490 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 02:49:11.409966 388 mini_ranger.cc:66] Stopping Ranger...
I20260929 02:49:11.756392 388 mini_ranger.cc:68] Stopped Ranger
2026-09-29 02:49:11.756 UTC [12282] LOG: received smart shutdown request
2026-09-29 02:49:11.760 UTC [12282] LOG: background worker "logical replication launcher" (PID 12291) exited with exit code 1
2026-09-29 02:49:11.761 UTC [12286] LOG: shutting down
2026-09-29 02:49:11.967 UTC [12282] LOG: database system is shut down
2026-09-29T02:49:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55345 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:12 dist-test-slave-6tjh krb5kdc[13812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:12 dist-test-slave-6tjh krb5kdc[13812](info): set up 2 sockets
Sep 29 02:49:12 dist-test-slave-6tjh krb5kdc[13812](info): commencing operation
krb5kdc: starting...
W20260929 02:49:13.237133 388 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:49:13 dist-test-slave-6tjh krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:49:13.477735 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38101
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38101,127.0.97.61:44609,127.0.97.60:33083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:13.821074 13825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:13.821408 13825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:13.821504 13825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:13.829036 13825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:13.829166 13825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:13.829276 13825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:13.829375 13825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:13.829443 13825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:13.839558 13825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38101,127.0.97.61:44609,127.0.97.60:33083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:13.840951 13825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:13.842275 13825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:13.855119 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:13.855232 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:13.855232 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:13.856511 13825 server_base.cc:1034] running on GCE node
I20260929 02:49:13.857501 13825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:13.859048 13825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:13.860270 13825 hybrid_clock.cc:648] HybridClock initialized: now 1790650153860219 us; error 44 us; skew 500 ppm
I20260929 02:49:13.860684 13825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:13 dist-test-slave-6tjh krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:13.870219 13825 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:13.871980 13825 webserver.cc:460] Webserver started at http://127.0.97.62:36333/ using document root <none> and password file <none>
I20260929 02:49:13.872794 13825 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:13.872907 13825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:13.873184 13825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:13.876369 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "2ca65ef0e2ef467a91d454dab296abf7"
format_stamp: "Formatted at 2026-09-29 02:49:13 on dist-test-slave-6tjh"
server_key: "462332298ca6f13bfd967c0568a785a1"
server_key_iv: "4fbf1ec80360d9ceff72ae01714b4abb"
server_key_version: "encryptionkey@0"
I20260929 02:49:13.877302 13825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "2ca65ef0e2ef467a91d454dab296abf7"
format_stamp: "Formatted at 2026-09-29 02:49:13 on dist-test-slave-6tjh"
server_key: "462332298ca6f13bfd967c0568a785a1"
server_key_iv: "4fbf1ec80360d9ceff72ae01714b4abb"
server_key_version: "encryptionkey@0"
I20260929 02:49:13.883116 13825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:49:13.887282 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:13.889128 13825 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:49:13.889310 13825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "2ca65ef0e2ef467a91d454dab296abf7"
format_stamp: "Formatted at 2026-09-29 02:49:13 on dist-test-slave-6tjh"
server_key: "462332298ca6f13bfd967c0568a785a1"
server_key_iv: "4fbf1ec80360d9ceff72ae01714b4abb"
server_key_version: "encryptionkey@0"
I20260929 02:49:13.889510 13825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:13.927078 13825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:13.932029 13825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:13.932296 13825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:13.952808 13825 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38101
I20260929 02:49:13.952845 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38101 every 8 connection(s)
I20260929 02:49:13.954299 13825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:13.958153 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:13.962096 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13825
I20260929 02:49:13.962320 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:49:13.962971 388 external_mini_cluster.cc:1428] Setting key 6c091803a68cdb11d7bc562f428daf8b
I20260929 02:49:13.965482 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:13 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 02:49:13.990082 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.968985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38101 (local address 127.0.0.1:49818)
0929 02:49:13.970360 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.970417 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:13.972268 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:13.973233 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:13.973240 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:13.974021 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:13.974362 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.974381 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.975927 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.975934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.977115 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:13.977128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.978084 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.978091 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:13.978252 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.978948 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:13.978982 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:13.981751 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:13.985404 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.985417 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.985427 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.985952 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.986434 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:13.986438 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:13.986441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:13.986762 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:13.987906 (+ 1144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:13.987913 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:13.988146 (+ 233us) client_negotiation.cc:769] Sending connection context
0929 02:49:13.988374 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 02:49:13.988915 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"spinlock_wait_cycles":29568,"thread_start_us":804,"threads_started":1}
I20260929 02:49:13.990124 13895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.968985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.0.97.62:38101)
0929 02:49:13.969981 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:13.970002 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:13.970578 (+ 576us) server_negotiation.cc:407] Connection header received
0929 02:49:13.972734 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:13.972740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:13.973001 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:13.973230 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:13.974572 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.975760 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.977258 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:13.977839 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:13.978053 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:13.982056 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:13.982065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:13.982079 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:13.982112 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:13.985225 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.986116 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.986124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.986126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.986281 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:13.986993 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:13.986997 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:13.986999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:13.987741 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:13.987999 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:13.988603 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 02:49:13.989150 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":707,"thread_start_us":441,"threads_started":1}
I20260929 02:49:13.993920 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:49:13.995328 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.994756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38092)
0929 02:49:13.995006 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.995164 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 02:49:13.995760 13841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
I20260929 02:49:13.997642 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:13.997248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38096)
0929 02:49:13.997401 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:13.997479 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:49:13.998040 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44609: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 02:49:14.018002 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44609
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38101,127.0.97.61:44609,127.0.97.60:33083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:14.047457 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } attempt: 1
I20260929 02:49:14.048847 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.048271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38106)
0929 02:49:14.048519 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.048659 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 02:49:14.049978 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.049569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38112)
0929 02:49:14.049739 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.049828 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260929 02:49:14.050177 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44609: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
I20260929 02:49:14.118430 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } attempt: 2
I20260929 02:49:14.119784 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.119187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38118)
0929 02:49:14.119484 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.119599 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 02:49:14.120965 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.120508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38132)
0929 02:49:14.120717 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.120823 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260929 02:49:14.121186 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44609: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
I20260929 02:49:14.212504 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } attempt: 3
I20260929 02:49:14.213881 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.213378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38136)
0929 02:49:14.213600 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.213695 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 02:49:14.215062 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.214646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38150)
0929 02:49:14.214828 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.214917 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 02:49:14.215246 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44609: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
I20260929 02:49:14.358526 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } attempt: 4
I20260929 02:49:14.359989 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.359373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38842)
0929 02:49:14.359730 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.359840 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260929 02:49:14.361428 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.360983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38854)
0929 02:49:14.361170 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.361274 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 02:49:14.361629 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:44609: Network error: Client connection negotiation failed: client connection to 127.0.97.61:44609: connect: Connection refused (error 111)
W20260929 02:49:14.377666 13899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:14.378041 13899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:14.378216 13899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:14.385881 13899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:14.386040 13899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:14.386118 13899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:14.386231 13899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:14.386320 13899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:14.396677 13899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38101,127.0.97.61:44609,127.0.97.60:33083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:44609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:14.398113 13899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:14.399454 13899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:14.412101 13904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:14.412065 13905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:14.412031 13907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:14.412981 13899 server_base.cc:1034] running on GCE node
I20260929 02:49:14.413852 13899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:14.415390 13899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:14.416652 13899 hybrid_clock.cc:648] HybridClock initialized: now 1790650154416611 us; error 38 us; skew 500 ppm
I20260929 02:49:14.417038 13899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:14 dist-test-slave-6tjh krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:14.420373 13899 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 02:49:14.422060 13899 webserver.cc:460] Webserver started at http://127.0.97.61:46673/ using document root <none> and password file <none>
I20260929 02:49:14.422801 13899 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:14.422909 13899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:14.423214 13899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:14.426507 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data/instance:
uuid: "3acd8e0cf6264fe297f0f77c86dbec75"
format_stamp: "Formatted at 2026-09-29 02:49:14 on dist-test-slave-6tjh"
server_key: "8b12a3da09005764492f16dd3b8e8d73"
server_key_iv: "6bf210287fcf3d721769c852b6d22c4a"
server_key_version: "encryptionkey@0"
I20260929 02:49:14.427407 13899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/wal/instance:
uuid: "3acd8e0cf6264fe297f0f77c86dbec75"
format_stamp: "Formatted at 2026-09-29 02:49:14 on dist-test-slave-6tjh"
server_key: "8b12a3da09005764492f16dd3b8e8d73"
server_key_iv: "6bf210287fcf3d721769c852b6d22c4a"
server_key_version: "encryptionkey@0"
I20260929 02:49:14.433161 13899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:49:14.437077 13914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:14.438654 13899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:14.438880 13899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/wal
uuid: "3acd8e0cf6264fe297f0f77c86dbec75"
format_stamp: "Formatted at 2026-09-29 02:49:14 on dist-test-slave-6tjh"
server_key: "8b12a3da09005764492f16dd3b8e8d73"
server_key_iv: "6bf210287fcf3d721769c852b6d22c4a"
server_key_version: "encryptionkey@0"
I20260929 02:49:14.439113 13899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:14.459054 13899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:14.464200 13899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:14.464596 13899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:14.483866 13899 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:44609
I20260929 02:49:14.483888 13965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:44609 every 8 connection(s)
I20260929 02:49:14.485414 13899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
I20260929 02:49:14.489439 13966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:14.492026 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13899
I20260929 02:49:14.492237 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-1/wal/instance
I20260929 02:49:14.492740 388 external_mini_cluster.cc:1428] Setting key a13889f0232a7d4e63053cf711a4a759
I20260929 02:49:14.496668 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Sep 29 02:49:14 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 02:49:14.520751 13969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.499704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.0.97.62:38101)
0929 02:49:14.501662 (+ 1958us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:14.501671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:14.502157 (+ 486us) server_negotiation.cc:407] Connection header received
0929 02:49:14.504159 (+ 2002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:14.504165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:14.504264 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:14.504393 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:14.506302 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.507115 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.508871 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.509300 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.509454 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:14.513569 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:14.513575 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:14.513579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:14.513626 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:14.516869 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:14.518016 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:14.518021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:14.518023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:14.518154 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:14.518688 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:14.518692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:14.518694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:14.519015 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:14.519141 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:14.520178 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0929 02:49:14.520362 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":1669,"threads_started":1}
I20260929 02:49:14.522408 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.500379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38101 (local address 127.0.0.1:44876)
0929 02:49:14.501978 (+ 1599us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.502020 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:14.503833 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:14.504795 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:14.504805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:14.505414 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:14.506053 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:14.506073 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.507310 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.507314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:14.508686 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:14.508701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.509577 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.509585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:14.509826 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:14.510568 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:14.510593 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:14.513195 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:14.517064 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:14.517078 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:14.517101 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:14.517828 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:14.518278 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:14.518287 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:14.518293 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:14.518564 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:14.519154 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:14.519166 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:14.519721 (+ 555us) client_negotiation.cc:769] Sending connection context
0929 02:49:14.520053 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 02:49:14.520675 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":1116,"threads_started":1}
I20260929 02:49:14.524708 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:14 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:49:14.543167 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.525587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38862)
0929 02:49:14.525848 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.525873 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:14.526011 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:14.527121 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:14.527129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:14.527769 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:14.528010 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:14.528026 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.529692 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.529701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:14.531414 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:14.531447 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.532457 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.532467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:14.532771 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:14.533488 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:14.533519 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:14.536826 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:14.540273 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:14.540277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:14.540279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:14.540791 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:14.541285 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:14.541293 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:14.541300 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:14.541476 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:14.542217 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:14.542221 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:14.542353 (+ 132us) client_negotiation.cc:769] Sending connection context
0929 02:49:14.542478 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 02:49:14.542820 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:49:14.543474 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.525845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.0.97.61:44609)
0929 02:49:14.526618 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:14.526626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:14.526654 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:49:14.526704 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:14.526709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:14.526921 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:14.527055 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:14.528217 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.529492 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.531618 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.532259 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.532384 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:14.537077 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:14.537091 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:14.537108 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:14.537143 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:14.540106 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:14.540963 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:14.540971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:14.540973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:14.541142 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:14.541604 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:14.541608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:14.541611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:14.542102 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:14.542285 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:14.542665 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 02:49:14.543028 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":597,"thread_start_us":421,"threads_started":1}
I20260929 02:49:14.546111 13966 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:49:14.547397 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.546874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42466)
0929 02:49:14.547102 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.547220 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260929 02:49:14.547683 13915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 02:49:14.549499 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.549127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42470)
0929 02:49:14.549296 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.549378 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 02:49:14.549481 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33083
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38101,127.0.97.61:44609,127.0.97.60:33083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:14.549878 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.599213 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 1
I20260929 02:49:14.600656 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.600036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42486)
0929 02:49:14.600288 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.600407 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 02:49:14.601765 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.601396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42492)
0929 02:49:14.601560 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.601655 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:49:14.602015 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.660898 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } attempt: 5
Sep 29 02:49:14 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:49:14.670248 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 2
I20260929 02:49:14.671293 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.670835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42506)
0929 02:49:14.671043 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.671137 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:49:14.672675 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.672103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42508)
0929 02:49:14.672306 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.672413 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 02:49:14.673095 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.676957 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.661997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.0.97.61:44609)
0929 02:49:14.662155 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:14.662160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:14.662180 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:49:14.662301 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:14.662305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:14.662367 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:14.662459 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:14.663446 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.664150 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.665905 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.666390 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.666505 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:14.670603 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:14.670612 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:14.670618 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:14.670674 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:14.674113 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:14.674887 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:14.674893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:14.674898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:14.675034 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:14.675446 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:14.675454 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:14.675457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:14.675762 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:14.675962 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:14.676334 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 02:49:14.676604 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 02:49:14.676964 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.661755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38866)
0929 02:49:14.661987 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.662012 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:14.662161 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:14.662526 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:14.662531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:14.663033 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:14.663259 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:14.663272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.664316 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.664322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:14.665703 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:14.665720 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:14.666562 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:14.666568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:14.666774 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:14.667467 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:14.667498 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:14.670279 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:14.674290 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:14.674297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:14.674301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:14.674744 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:14.675161 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:14.675166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:14.675169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:14.675326 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:14.675892 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:14.675897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:14.676072 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 02:49:14.676222 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 02:49:14.676603 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:49:14.678092 13892 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:49:14.679234 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.678833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42510)
0929 02:49:14.679001 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.679093 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:49:14.680462 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.680101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42524)
0929 02:49:14.680260 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.680337 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 02:49:14.680745 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.732013 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 1
I20260929 02:49:14.733397 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.732885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42538)
0929 02:49:14.733112 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.733206 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260929 02:49:14.734597 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.734238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42552)
0929 02:49:14.734405 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.734477 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 02:49:14.734898 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.764389 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 3
I20260929 02:49:14.765755 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.765177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42562)
0929 02:49:14.765429 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.765544 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 02:49:14.767000 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.766486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42578)
0929 02:49:14.766729 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.766830 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260929 02:49:14.767241 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.803174 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 2
I20260929 02:49:14.804355 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.803866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42584)
0929 02:49:14.804096 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.804193 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:49:14.805456 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.805096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42590)
0929 02:49:14.805260 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.805344 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:49:14.805794 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.910506 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 4
I20260929 02:49:14.911948 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.911265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42606)
0929 02:49:14.911547 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.911656 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 02:49:14.912052 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 3
I20260929 02:49:14.913386 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.912833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42616)
0929 02:49:14.913080 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.913186 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260929 02:49:14.913646 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.913187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42628)
0929 02:49:14.913393 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.913493 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260929 02:49:14.913921 13966 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
I20260929 02:49:14.914601 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:14.914242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42632)
0929 02:49:14.914429 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:14.914503 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:49:14.914805 13892 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:33083: Network error: Client connection negotiation failed: client connection to 127.0.97.60:33083: connect: Connection refused (error 111)
W20260929 02:49:14.921187 13974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:14.921509 13974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:14.921602 13974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:14.930059 13974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:14.930195 13974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:14.930243 13974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:14.930298 13974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:14.930342 13974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:14.941157 13974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38101,127.0.97.61:44609,127.0.97.60:33083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:33083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:14.942606 13974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:14.944226 13974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:14.959086 13979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:14.959086 13980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:14.959676 13982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:14.960673 13974 server_base.cc:1034] running on GCE node
I20260929 02:49:14.961733 13974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:14.963644 13974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:14.964876 13974 hybrid_clock.cc:648] HybridClock initialized: now 1790650154964817 us; error 64 us; skew 500 ppm
I20260929 02:49:14.965314 13974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:14 dist-test-slave-6tjh krb5kdc[13812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:14.968834 13974 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 02:49:14.970595 13974 webserver.cc:460] Webserver started at http://127.0.97.60:40109/ using document root <none> and password file <none>
I20260929 02:49:14.971406 13974 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:14.971557 13974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:14.971923 13974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:14.975282 13974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data/instance:
uuid: "fc88c5b3d8b24e45a541e7b7e9408052"
format_stamp: "Formatted at 2026-09-29 02:49:14 on dist-test-slave-6tjh"
server_key: "a1b0b91b8379c41d7acea2dcdb5a8028"
server_key_iv: "8f9e953e4cfa967d0a4ad412ee414a06"
server_key_version: "encryptionkey@0"
I20260929 02:49:14.976169 13974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/wal/instance:
uuid: "fc88c5b3d8b24e45a541e7b7e9408052"
format_stamp: "Formatted at 2026-09-29 02:49:14 on dist-test-slave-6tjh"
server_key: "a1b0b91b8379c41d7acea2dcdb5a8028"
server_key_iv: "8f9e953e4cfa967d0a4ad412ee414a06"
server_key_version: "encryptionkey@0"
I20260929 02:49:14.982095 13974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:49:14.986441 13989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:14.988131 13974 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:49:14.988350 13974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/wal
uuid: "fc88c5b3d8b24e45a541e7b7e9408052"
format_stamp: "Formatted at 2026-09-29 02:49:14 on dist-test-slave-6tjh"
server_key: "a1b0b91b8379c41d7acea2dcdb5a8028"
server_key_iv: "8f9e953e4cfa967d0a4ad412ee414a06"
server_key_version: "encryptionkey@0"
I20260929 02:49:14.988663 13974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:15.022214 13974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:15.027590 13974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:15.027863 13974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:15.047456 13974 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:33083
I20260929 02:49:15.047487 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:33083 every 8 connection(s)
I20260929 02:49:15.049096 13974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
I20260929 02:49:15.053261 14041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:15.054123 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 13974
I20260929 02:49:15.054301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790649949495923-388-0/minicluster-data/master-2/wal/instance
I20260929 02:49:15.054796 388 external_mini_cluster.cc:1428] Setting key 8b9a9331a953ee3750e488f6f170aa02
I20260929 02:49:15.063987 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:15.082490 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.056874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.0.97.60:33083)
0929 02:49:15.057713 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.057731 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.057790 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:49:15.060470 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.060517 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.061196 (+ 679us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.061749 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.063245 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.065341 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.067089 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.068236 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.069245 (+ 1009us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.073369 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.073381 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.073422 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.073486 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.077500 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.078471 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.078481 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.078493 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.078717 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.079245 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.079253 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.079256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.079851 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.080164 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.080698 (+ 534us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.081154 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":350,"threads_started":1}
I20260929 02:49:15.084869 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.067179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38101 (local address 127.0.0.1:44890)
0929 02:49:15.068005 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:15.068056 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:15.068236 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:15.069192 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:15.069202 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:15.069962 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:15.070224 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.070234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.071299 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.071312 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.072797 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.072819 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.074028 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.074034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.074174 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.074899 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:15.074926 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:15.077815 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:15.081823 (+ 4008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.081833 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.081840 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.082450 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.083009 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.083014 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.083017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.083226 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.083818 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:15.083826 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:15.084045 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 02:49:15.084224 (+ 179us) client_negotiation.cc:240] Negotiation successful
0929 02:49:15.084504 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":344,"threads_started":1}
I20260929 02:49:15.085098 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.067295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44890 (local address 127.0.97.62:38101)
0929 02:49:15.068755 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.068761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.068780 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:49:15.068840 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.068847 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.068947 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.069075 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.070411 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.071127 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.073024 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.073762 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.073930 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.078075 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.078081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.078085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.078136 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.081660 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.082654 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.082661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.082667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.082876 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.083389 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.083393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.083395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.083685 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.083829 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.084431 (+ 602us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.084728 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1329,"thread_start_us":1157,"threads_started":1}
I20260929 02:49:15.087163 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:49:15.092067 13892 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 4
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 02:49:15.103920 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.088049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:44609 (local address 127.0.0.1:38882)
0929 02:49:15.088255 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:15.088277 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:15.088419 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:15.088823 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:15.088829 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:15.089316 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:15.089535 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.089561 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.090734 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.090742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.091640 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.091682 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.092677 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.092684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.093037 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.093732 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:15.093751 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:15.096569 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:15.101068 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.101073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.101076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.101604 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.102058 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.102063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.102067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.102236 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.102806 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:15.102812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:15.103002 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 02:49:15.103163 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 02:49:15.103524 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 02:49:15.104874 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.088118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.0.97.61:44609)
0929 02:49:15.088317 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.088321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.088343 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:49:15.088599 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.088604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.088675 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.088756 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.089712 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.090507 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.091828 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.092284 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.092417 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.096834 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.096843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.096848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.096904 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.100891 (+ 3987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.101746 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.101752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.101756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.101925 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.102358 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.102363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.102367 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.102671 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.102852 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.103567 (+ 715us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.103824 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:49:15.105954 14041 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:49:15.107793 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.086918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.97.62:38101)
0929 02:49:15.087092 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.087098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.087117 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:49:15.087185 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.087192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.087280 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.087419 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.089384 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.090246 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.091744 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.092461 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.094465 (+ 2004us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.098482 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.098489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.098494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.098553 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.103119 (+ 4566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.104411 (+ 1292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.104419 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.104424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.104637 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.105167 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.105176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.105180 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.105577 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.105889 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.107235 (+ 1346us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.107554 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:15.109690 13896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.092764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42658)
0929 02:49:15.093039 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:15.093060 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:15.093166 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:15.094156 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:15.094161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:15.094682 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:15.094892 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.094906 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.096441 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.096448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.097579 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.097595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.098603 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.098610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.098815 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.099474 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:15.099503 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:15.102224 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:15.106446 (+ 4222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.106450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.106454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.106988 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.107507 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.107513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.107516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.107709 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.108599 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:15.108604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:15.108773 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 02:49:15.108926 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:49:15.109287 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260929 02:49:15.110826 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.093022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.97.60:33083)
0929 02:49:15.093758 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.093763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.093783 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:49:15.093836 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.093840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.093926 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.094108 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.095080 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.096260 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.097752 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.098421 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.098583 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.102480 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.102488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.102492 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.102533 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.106303 (+ 3770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.107136 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.107141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.107144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.107341 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.108014 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.108019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.108021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.108420 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.110090 (+ 1670us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.110210 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.110466 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 02:49:15.116508 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7: Bootstrap starting.
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:49:15.120801 13892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:15.122195 13892 log.cc:824] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:15.124790 14045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.106690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42666)
0929 02:49:15.106994 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:15.107019 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:15.107130 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:15.108029 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:15.108034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:15.108594 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:15.108875 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.108890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.110666 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.110672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.112237 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.112251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.113233 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.113239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.113456 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.114145 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:15.114168 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:15.117367 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:15.121532 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.121539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.121544 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.122058 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.122568 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.122576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.122582 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.122746 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.123466 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:15.123470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:15.123643 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 02:49:15.123816 (+ 173us) client_negotiation.cc:240] Negotiation successful
0929 02:49:15.124163 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"spinlock_wait_cycles":25984}
I20260929 02:49:15.124859 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.106903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.97.60:33083)
0929 02:49:15.107714 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.107720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.107742 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:49:15.107793 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.107800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.107879 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.108634 (+ 755us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.109064 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.110531 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.112386 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.113039 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.113192 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.117623 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.117629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.117634 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.117694 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.121358 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.122195 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.122201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.122204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.122426 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.122880 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.122885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.122888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.123258 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.123381 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.123952 (+ 571us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.124335 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":685,"spinlock_wait_cycles":40320,"thread_start_us":450,"threads_started":1}
I20260929 02:49:15.125656 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7: No bootstrap required, opened a new log
I20260929 02:49:15.127440 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.109755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38890 (local address 127.0.97.61:44609)
0929 02:49:15.110095 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.110101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.110120 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:49:15.110218 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.110224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.110296 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.110404 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.112032 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.113005 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.114776 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.115478 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.115587 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.120697 (+ 5110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.120702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.120706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.120742 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.124235 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.125227 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.125233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.125236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.125395 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.125828 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.125833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.125835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.126150 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.126337 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.126844 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.127084 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260929 02:49:15.131328 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: Bootstrap starting.
I20260929 02:49:15.132411 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:49:15.135561 14041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:15.136998 14041 log.cc:824] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:15.136976 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.137398 13892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:15.137528 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca65ef0e2ef467a91d454dab296abf7, State: Initialized, Role: FOLLOWER
I20260929 02:49:15.138463 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.142731 14055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.143265 14055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:15.145833 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:15.151424 14041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: No bootstrap required, opened a new log
I20260929 02:49:15.165470 14046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.139338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.0.97.62:38101)
0929 02:49:15.139529 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.139536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.140305 (+ 769us) server_negotiation.cc:407] Connection header received
0929 02:49:15.140661 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.140670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.140793 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.140888 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.144538 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.146147 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.149118 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.150168 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.150336 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.156054 (+ 5718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.156065 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.156073 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.156157 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.160449 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.161389 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.161395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.161399 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.161575 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.162098 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.162105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.162109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.163737 (+ 1628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.164089 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.164763 (+ 674us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.165045 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:49:15.166131 14041 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.166676 14041 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:15.166843 14041 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc88c5b3d8b24e45a541e7b7e9408052, State: Initialized, Role: FOLLOWER
I20260929 02:49:15.168151 14041 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.172747 13972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.139345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.0.97.61:44609)
0929 02:49:15.139520 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.139527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.152228 (+ 12701us) server_negotiation.cc:407] Connection header received
0929 02:49:15.153617 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.153625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.153719 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.153802 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.156741 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.157948 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.160248 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.160936 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.161053 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.165086 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.165093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.165097 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.165143 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.169000 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.169947 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.169955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.169960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.170149 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.170654 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.170661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.170665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.171149 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.171409 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.172042 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.172315 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:15.172703 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.173316 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:15.175866 14041 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:15.179288 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.152853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.97.60:33083)
0929 02:49:15.153085 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.153095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.157161 (+ 4066us) server_negotiation.cc:407] Connection header received
0929 02:49:15.157352 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.157361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.157488 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.157610 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.161660 (+ 4050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.163010 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.167512 (+ 4502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.168180 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.168296 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.172306 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.172313 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.172318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.172359 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.176141 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.177016 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.177021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.177025 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.177184 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.177668 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.177673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.177677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.177992 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.178215 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.178681 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.178910 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260929 02:49:15.185621 13846 tablet.cc:2369] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:15.201479 14080 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 02:49:15.202292 14079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:49:15.202469 14079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:49:15.192152 13974 master_runner.cc:186] Error getting master registration for 127.0.97.62:38101: OK, instance_id { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" instance_seqno: 1790650153946880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:49:15.206733 13974 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:49:15.213189 13966 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } attempt: 5
I20260929 02:49:15.206135 13825 master_runner.cc:186] Error getting master registration for 127.0.97.61:44609: OK, instance_id { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" instance_seqno: 1790650154478580 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:49:15.214401 13825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 29 02:49:15 dist-test-slave-6tjh krb5kdc[13812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 02:49:15.232161 14047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.214109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.0.97.60:33083)
0929 02:49:15.214297 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:15.214305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:15.214430 (+ 125us) server_negotiation.cc:407] Connection header received
0929 02:49:15.214599 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:15.214604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:15.214714 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:15.214812 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:15.215888 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.216927 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.218898 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.219519 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.219787 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.224656 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:15.224665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:15.224668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:15.224707 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:15.227771 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.228932 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.228936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.228939 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.229257 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:15.230336 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:15.230342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:15.230345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:15.230784 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:15.230954 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:15.231468 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 02:49:15.231740 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:49:15.233816 13970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:15.213981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:33083 (local address 127.0.0.1:42676)
0929 02:49:15.214293 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:15.214316 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:15.214464 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:15.214901 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:15.214906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:15.215461 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:15.215680 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.215693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.217092 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.217098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.218645 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:15.218670 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:15.219815 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:15.219821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:15.220037 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:15.221402 (+ 1365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:15.221439 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:15.224297 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:15.228017 (+ 3720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.228024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.228028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.228724 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.229894 (+ 1170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:15.229901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:15.229916 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:15.230129 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:15.230945 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:15.230950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:15.231140 (+ 190us) client_negotiation.cc:769] Sending connection context
0929 02:49:15.231315 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 02:49:15.231674 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260929 02:49:15.238921 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: Bootstrap starting.
I20260929 02:49:15.242372 13966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:15.243793 13966 log.cc:824] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:15.247709 13966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: No bootstrap required, opened a new log
I20260929 02:49:15.255913 13966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.256417 13966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:15.256562 13966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acd8e0cf6264fe297f0f77c86dbec75, State: Initialized, Role: FOLLOWER
I20260929 02:49:15.257372 13966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.258459 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.258912 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:15.259719 13966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:15.277665 13899 master_runner.cc:263] No leader master found from master 3acd8e0cf6264fe297f0f77c86dbec75
I20260929 02:49:15.277878 13899 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 02:49:15.280359 13995 tablet.cc:2369] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:49:15.283653 14094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:49:15.283835 14094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:49:15.363202 14055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:15.363577 14055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.365381 14055 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3acd8e0cf6264fe297f0f77c86dbec75 (127.0.97.61:44609), fc88c5b3d8b24e45a541e7b7e9408052 (127.0.97.60:33083)
I20260929 02:49:15.366324 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ca65ef0e2ef467a91d454dab296abf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" is_pre_election: true
I20260929 02:49:15.366328 14016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ca65ef0e2ef467a91d454dab296abf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" is_pre_election: true
I20260929 02:49:15.366983 14016 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca65ef0e2ef467a91d454dab296abf7 in term 0.
I20260929 02:49:15.366979 13941 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ca65ef0e2ef467a91d454dab296abf7 in term 0.
I20260929 02:49:15.367623 13841 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca65ef0e2ef467a91d454dab296abf7, fc88c5b3d8b24e45a541e7b7e9408052; no voters:
I20260929 02:49:15.368098 14055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:15.368257 14055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:15.368363 14055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:15.371836 14055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.372700 14055 leader_election.cc:288] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [CANDIDATE]: Term 1 election: Requested vote from peers 3acd8e0cf6264fe297f0f77c86dbec75 (127.0.97.61:44609), fc88c5b3d8b24e45a541e7b7e9408052 (127.0.97.60:33083)
I20260929 02:49:15.373415 14016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ca65ef0e2ef467a91d454dab296abf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc88c5b3d8b24e45a541e7b7e9408052"
I20260929 02:49:15.373450 13941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ca65ef0e2ef467a91d454dab296abf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acd8e0cf6264fe297f0f77c86dbec75"
I20260929 02:49:15.373661 14016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:15.373695 13941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:15.377354 14016 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca65ef0e2ef467a91d454dab296abf7 in term 1.
I20260929 02:49:15.377359 13941 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca65ef0e2ef467a91d454dab296abf7 in term 1.
I20260929 02:49:15.378011 13841 leader_election.cc:302] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca65ef0e2ef467a91d454dab296abf7, 3acd8e0cf6264fe297f0f77c86dbec75; no voters:
I20260929 02:49:15.378381 14055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:15.378996 14055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [term 1 LEADER]: Becoming Leader. State: Replica: 2ca65ef0e2ef467a91d454dab296abf7, State: Running, Role: LEADER
I20260929 02:49:15.381873 14055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } }
I20260929 02:49:15.386394 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca65ef0e2ef467a91d454dab296abf7. Latest consensus state: current_term: 1 leader_uuid: "2ca65ef0e2ef467a91d454dab296abf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.386904 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:15.397315 14098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:15.403358 14098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:15.416172 14016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [term 1 FOLLOWER]: Refusing update from remote peer 2ca65ef0e2ef467a91d454dab296abf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:15.417379 14096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:15.425149 13941 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [term 1 FOLLOWER]: Refusing update from remote peer 2ca65ef0e2ef467a91d454dab296abf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:15.426222 14096 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:15.446693 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca65ef0e2ef467a91d454dab296abf7. Latest consensus state: current_term: 1 leader_uuid: "2ca65ef0e2ef467a91d454dab296abf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.447198 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:15.454864 14100 mvcc.cc:205] Tried to move back new op lower bound from 7334503036571406336 to 7334503036451688448. Current Snapshot: MvccSnapshot[applied={T|T < 7334503036571406336}]
I20260929 02:49:15.459899 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ca65ef0e2ef467a91d454dab296abf7. Latest consensus state: current_term: 1 leader_uuid: "2ca65ef0e2ef467a91d454dab296abf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.460278 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:15.465462 14102 mvcc.cc:205] Tried to move back new op lower bound from 7334503036571406336 to 7334503036451688448. Current Snapshot: MvccSnapshot[applied={T|T < 7334503036571406336}]
I20260929 02:49:15.466157 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ca65ef0e2ef467a91d454dab296abf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.466548 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:15.467092 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ca65ef0e2ef467a91d454dab296abf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.467383 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:15.468497 14098 catalog_manager.cc:1347] Generated new cluster ID: 1fe93126fb194e98a24986a5f1d916f4
I20260929 02:49:15.468641 14098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:15.469679 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ca65ef0e2ef467a91d454dab296abf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.469950 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:15.470350 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ca65ef0e2ef467a91d454dab296abf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ca65ef0e2ef467a91d454dab296abf7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38101 } } peers { permanent_uuid: "3acd8e0cf6264fe297f0f77c86dbec75" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 44609 } } peers { permanent_uuid: "fc88c5b3d8b24e45a541e7b7e9408052" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 33083 } } }
I20260929 02:49:15.470640 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:49:15.493197 14098 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:15.495273 14098 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:15.514185 14098 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ca65ef0e2ef467a91d454dab296abf7: Generated new TSK 0
I20260929 02:49:15.515399 14098 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:49:16.204437 14079 catalog_manager.cc:1259] Loaded cluster ID: 1fe93126fb194e98a24986a5f1d916f4
I20260929 02:49:16.204581 14079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: loading cluster ID for follower catalog manager: success
I20260929 02:49:16.207029 14079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: acquiring CA information for follower catalog manager: success
I20260929 02:49:16.208958 14079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fc88c5b3d8b24e45a541e7b7e9408052: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:49:16.286084 14094 catalog_manager.cc:1259] Loaded cluster ID: 1fe93126fb194e98a24986a5f1d916f4
I20260929 02:49:16.286196 14094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: loading cluster ID for follower catalog manager: success
I20260929 02:49:16.288625 14094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: acquiring CA information for follower catalog manager: success
I20260929 02:49:16.290452 14094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3acd8e0cf6264fe297f0f77c86dbec75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:49:16.662572 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13825
I20260929 02:49:16.664202 13887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:16.780694 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13825
I20260929 02:49:16.802501 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13899
I20260929 02:49:16.803783 13961 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:16.915916 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13899
I20260929 02:49:16.937072 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 13974
I20260929 02:49:16.938303 14036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:17.049817 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 13974
2026-09-29T02:49:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4951 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:17 dist-test-slave-6tjh krb5kdc[14117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:17 dist-test-slave-6tjh krb5kdc[14117](info): set up 2 sockets
Sep 29 02:49:17 dist-test-slave-6tjh krb5kdc[14117](info): commencing operation
krb5kdc: starting...
W20260929 02:49:18.221158 388 mini_kdc.cc:121] Time spent starting KDC: real 1.133s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:49:18 dist-test-slave-6tjh krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:49:18.481612 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42611
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:18.854952 14130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:18.855289 14130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:18.855476 14130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:18.863399 14130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:18.863547 14130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:18.863595 14130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:18.863739 14130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:18.863843 14130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:18.875722 14130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:18.877172 14130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:18.878623 14130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:18.891800 14138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:18.891851 14135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:18.891880 14136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:18.892827 14130 server_base.cc:1034] running on GCE node
I20260929 02:49:18.893827 14130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:18.895490 14130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:18.896745 14130 hybrid_clock.cc:648] HybridClock initialized: now 1790650158896692 us; error 55 us; skew 500 ppm
I20260929 02:49:18.897195 14130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:18 dist-test-slave-6tjh krb5kdc[14117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650158, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:18.908495 14130 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:18.910339 14130 webserver.cc:460] Webserver started at http://127.0.97.62:40307/ using document root <none> and password file <none>
I20260929 02:49:18.911119 14130 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:18.911232 14130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:18.911504 14130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:18.914738 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "7b68ab83ade247c4938bb10d3b129e59"
format_stamp: "Formatted at 2026-09-29 02:49:18 on dist-test-slave-6tjh"
server_key: "9955b72dfb77dca5f9a06825224cf32a"
server_key_iv: "9b0649db778998c1c4edd43a5c46794e"
server_key_version: "encryptionkey@0"
I20260929 02:49:18.915529 14130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "7b68ab83ade247c4938bb10d3b129e59"
format_stamp: "Formatted at 2026-09-29 02:49:18 on dist-test-slave-6tjh"
server_key: "9955b72dfb77dca5f9a06825224cf32a"
server_key_iv: "9b0649db778998c1c4edd43a5c46794e"
server_key_version: "encryptionkey@0"
I20260929 02:49:18.921325 14130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:49:18.925146 14145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:18.926789 14130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:49:18.926996 14130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "7b68ab83ade247c4938bb10d3b129e59"
format_stamp: "Formatted at 2026-09-29 02:49:18 on dist-test-slave-6tjh"
server_key: "9955b72dfb77dca5f9a06825224cf32a"
server_key_iv: "9b0649db778998c1c4edd43a5c46794e"
server_key_version: "encryptionkey@0"
I20260929 02:49:18.927187 14130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:18.979424 14130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:18.984441 14130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:18.984751 14130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:19.006290 14130 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42611
I20260929 02:49:19.006291 14196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42611 every 8 connection(s)
I20260929 02:49:19.008038 14130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:19.008493 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 14130
I20260929 02:49:19.008744 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:49:19.009727 388 external_mini_cluster.cc:1428] Setting key b37f9d07d15df68fd38a420f0866d900
I20260929 02:49:19.013510 14197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:49:19 dist-test-slave-6tjh krb5kdc[14117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650158, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:19.025321 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59: Bootstrap starting.
I20260929 02:49:19.029839 14197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:19.031261 14197 log.cc:824] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:19.035233 14197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59: No bootstrap required, opened a new log
I20260929 02:49:19.036381 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:19.012005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.0.97.62:42611)
0929 02:49:19.013024 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:19.013042 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:19.013091 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:49:19.014972 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:19.014998 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:19.015433 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:19.016041 (+ 608us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:19.017727 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:19.019171 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:19.020616 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:19.021342 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:19.021554 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:19.026155 (+ 4601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:19.026167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:19.026189 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:19.026251 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:19.030535 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:19.031703 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:19.031712 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:19.031720 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:19.032012 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:19.032631 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:19.032637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:19.032640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:19.033403 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:19.033711 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:19.034345 (+ 634us) server_negotiation.cc:299] Negotiation successful
0929 02:49:19.034847 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":534,"threads_started":1}
I20260929 02:49:19.041128 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:49:19.044535 14197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b68ab83ade247c4938bb10d3b129e59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42611 } }
I20260929 02:49:19.045125 14197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:19.045258 14197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b68ab83ade247c4938bb10d3b129e59, State: Initialized, Role: FOLLOWER
I20260929 02:49:19.045950 14197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b68ab83ade247c4938bb10d3b129e59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42611 } }
I20260929 02:49:19.046186 14197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:19.046348 14197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:19.046545 14197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:19.049019 14197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b68ab83ade247c4938bb10d3b129e59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42611 } }
I20260929 02:49:19.049638 14197 leader_election.cc:302] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b68ab83ade247c4938bb10d3b129e59; no voters:
I20260929 02:49:19.050488 14197 leader_election.cc:288] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:19.050602 14202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:19.051440 14202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [term 1 LEADER]: Becoming Leader. State: Replica: 7b68ab83ade247c4938bb10d3b129e59, State: Running, Role: LEADER
I20260929 02:49:19.052131 14202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b68ab83ade247c4938bb10d3b129e59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42611 } }
I20260929 02:49:19.052968 14197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:19.055320 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b68ab83ade247c4938bb10d3b129e59. Latest consensus state: current_term: 1 leader_uuid: "7b68ab83ade247c4938bb10d3b129e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b68ab83ade247c4938bb10d3b129e59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42611 } } }
I20260929 02:49:19.055822 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:19.055589 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b68ab83ade247c4938bb10d3b129e59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b68ab83ade247c4938bb10d3b129e59" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42611 } } }
I20260929 02:49:19.055950 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:19.058019 14211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:19.063740 14211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:19.074851 14211 catalog_manager.cc:1347] Generated new cluster ID: df324371a65f46789e684b73d0191d34
I20260929 02:49:19.075008 14211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:19.102233 14211 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:19.103770 14211 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:19.112195 14211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b68ab83ade247c4938bb10d3b129e59: Generated new TSK 0
I20260929 02:49:19.113271 14211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:49:19.156665 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 14130
I20260929 02:49:19.158188 14192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:19.259094 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 14130
2026-09-29T02:49:19Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2209 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:19 dist-test-slave-6tjh krb5kdc[14225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:19 dist-test-slave-6tjh krb5kdc[14225](info): set up 2 sockets
Sep 29 02:49:19 dist-test-slave-6tjh krb5kdc[14225](info): commencing operation
krb5kdc: starting...
W20260929 02:49:20.401103 388 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:49:20 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:49:20.657640 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33279
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:21.009832 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:21.010334 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:21.010504 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:21.018254 14238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:21.018442 14238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:21.018560 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:21.018671 14238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:21.018759 14238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:21.029310 14238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:21.030854 14238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:21.032244 14238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:21.045034 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.045125 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.045125 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:21.046388 14238 server_base.cc:1034] running on GCE node
I20260929 02:49:21.047359 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:21.049068 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:21.050354 14238 hybrid_clock.cc:648] HybridClock initialized: now 1790650161050263 us; error 81 us; skew 500 ppm
I20260929 02:49:21.050854 14238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:21 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:21.054631 14238 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:21.056329 14238 webserver.cc:460] Webserver started at http://127.0.97.62:37755/ using document root <none> and password file <none>
I20260929 02:49:21.057191 14238 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:21.057356 14238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:21.057679 14238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:21.061107 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "c85173b927e446c9afdd21998d985120"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "7013927c8685e2511d41b8a7c9801bb2"
server_key_iv: "757648138be9290e3bea59a25ebc1d78"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.061967 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "c85173b927e446c9afdd21998d985120"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "7013927c8685e2511d41b8a7c9801bb2"
server_key_iv: "757648138be9290e3bea59a25ebc1d78"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.068203 14238 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 02:49:21.072376 14253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.074173 14238 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:49:21.074404 14238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "c85173b927e446c9afdd21998d985120"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "7013927c8685e2511d41b8a7c9801bb2"
server_key_iv: "757648138be9290e3bea59a25ebc1d78"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.074637 14238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:21.104360 14238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:21.109705 14238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:21.110080 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:21.132025 14238 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33279
I20260929 02:49:21.132056 14304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33279 every 8 connection(s)
I20260929 02:49:21.133705 14238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:21.138082 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:21.143057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 14238
I20260929 02:49:21.143337 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:49:21.144099 388 external_mini_cluster.cc:1428] Setting key 5a39b856acafc87b376b928de3aa3198
I20260929 02:49:21.152083 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Bootstrap starting.
I20260929 02:49:21.156433 14305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:21.157825 14305 log.cc:824] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:21.161623 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: No bootstrap required, opened a new log
Sep 29 02:49:21 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:21.173175 14305 raft_consensus.cc:348] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:21.173755 14305 raft_consensus.cc:374] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:21.173882 14305 raft_consensus.cc:729] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85173b927e446c9afdd21998d985120, State: Initialized, Role: FOLLOWER
I20260929 02:49:21.174371 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:21.146670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.97.62:33279)
0929 02:49:21.147667 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:21.147694 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:21.147764 (+ 70us) server_negotiation.cc:407] Connection header received
0929 02:49:21.150444 (+ 2680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:21.150481 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:21.151106 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:21.151672 (+ 566us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:21.153496 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.155654 (+ 2158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.157407 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.158668 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.158871 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:21.163859 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:21.163871 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:21.163895 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:21.163952 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:21.168584 (+ 4632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.169536 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.169558 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.169569 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.169852 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.170287 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.170292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.170296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.171228 (+ 932us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:21.171608 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:21.172101 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 02:49:21.172689 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":415,"threads_started":1}
I20260929 02:49:21.174652 14305 consensus_queue.cc:260] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:21.174963 14305 raft_consensus.cc:388] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:21.175139 14305 raft_consensus.cc:482] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:21.175338 14305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:21.177989 14305 raft_consensus.cc:504] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:21.178587 14305 leader_election.cc:302] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85173b927e446c9afdd21998d985120; no voters:
I20260929 02:49:21.179260 14305 leader_election.cc:288] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:21.179508 14310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:21.180382 14310 raft_consensus.cc:686] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 LEADER]: Becoming Leader. State: Replica: c85173b927e446c9afdd21998d985120, State: Running, Role: LEADER
I20260929 02:49:21.181109 14310 consensus_queue.cc:237] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:21.182026 14305 sys_catalog.cc:564] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:21.185249 14311 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c85173b927e446c9afdd21998d985120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } } }
I20260929 02:49:21.186160 14311 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:21.186784 14312 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c85173b927e446c9afdd21998d985120. Latest consensus state: current_term: 1 leader_uuid: "c85173b927e446c9afdd21998d985120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } } }
I20260929 02:49:21.189014 14312 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:21.191705 14318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:21.198760 14318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:21.209247 14318 catalog_manager.cc:1347] Generated new cluster ID: 5eaf707e6c5a466f9902821e77c7ed7a
I20260929 02:49:21.209399 14318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:21.232622 14318 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:21.234185 14318 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:21.244426 14318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Generated new TSK 0
I20260929 02:49:21.245528 14318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:49:21.312827 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33279
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:21.686390 14331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:21.686715 14331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:21.686872 14331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:21.694957 14331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:21.695102 14331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:21.695281 14331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:49:21.707558 14331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:21.709177 14331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:21.710695 14331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:21.726930 14337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.726958 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:21.727031 14339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:21.728427 14331 server_base.cc:1034] running on GCE node
I20260929 02:49:21.729205 14331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:21.730638 14331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:21.731990 14331 hybrid_clock.cc:648] HybridClock initialized: now 1790650161731870 us; error 127 us; skew 500 ppm
I20260929 02:49:21.732394 14331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:21 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:21.737085 14331 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:49:21.738726 14331 webserver.cc:460] Webserver started at http://127.0.97.1:46665/ using document root <none> and password file <none>
I20260929 02:49:21.739521 14331 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:21.739686 14331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:21.740033 14331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:21.743309 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "5344d6ef23db4e5a921786826cfd1099"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "882ec58f47d320d254ee3639bd8c3dca"
server_key_iv: "f04774914cc5576eb0362df002ab1869"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.744202 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5344d6ef23db4e5a921786826cfd1099"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "882ec58f47d320d254ee3639bd8c3dca"
server_key_iv: "f04774914cc5576eb0362df002ab1869"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.750115 14331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:49:21.754698 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.756402 14331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:49:21.756707 14331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "5344d6ef23db4e5a921786826cfd1099"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "882ec58f47d320d254ee3639bd8c3dca"
server_key_iv: "f04774914cc5576eb0362df002ab1869"
server_key_version: "encryptionkey@0"
I20260929 02:49:21.756976 14331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:21.783726 14331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:21.788638 14331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:21.789002 14331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:21.790161 14331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:21.791651 14331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:21.791764 14331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.791925 14331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:21.792001 14331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:21.828687 14331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41553
I20260929 02:49:21.828708 14458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41553 every 8 connection(s)
I20260929 02:49:21.830159 14331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:21.832393 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 14331
I20260929 02:49:21.832638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:21.833542 388 external_mini_cluster.cc:1428] Setting key a204efa56df90af87ec41c1397a617e0
Sep 29 02:49:21 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:21.858944 14461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:21.833266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57965 (local address 127.0.97.62:33279)
0929 02:49:21.833930 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:21.833938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:21.835070 (+ 1132us) server_negotiation.cc:407] Connection header received
0929 02:49:21.838145 (+ 3075us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:21.838150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:21.838224 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:21.838326 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:21.841023 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.841789 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.844896 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.845689 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.845866 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:21.850246 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:21.850256 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:21.850265 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:21.850322 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:21.854510 (+ 4188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.855642 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.855651 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.855658 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.855870 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:21.856466 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:21.856474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:21.856480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:21.856977 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:21.857161 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:21.858090 (+ 929us) server_negotiation.cc:299] Negotiation successful
0929 02:49:21.858428 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":285,"threads_started":1}
I20260929 02:49:21.859616 14462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:21.833898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:57965)
0929 02:49:21.834874 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:21.834948 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:21.837742 (+ 2794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:21.838667 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:21.838684 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:21.839634 (+ 950us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:21.840729 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:21.840767 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.842003 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.842013 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:21.844654 (+ 2641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:21.844681 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:21.846010 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:21.846016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:21.846266 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:21.847083 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:21.847109 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:21.849889 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:21.854770 (+ 4881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:21.854780 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:21.854797 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:21.855427 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:21.856058 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:21.856062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:21.856065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:21.856301 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:21.857149 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:21.857157 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:21.857617 (+ 460us) client_negotiation.cc:769] Sending connection context
0929 02:49:21.857874 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 02:49:21.858374 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":395,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:49:21.861812 14459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33279
I20260929 02:49:21.862243 14459 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:21.863161 14459 heartbeater.cc:511] Master 127.0.97.62:33279 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:49:21.865859 14270 ts_manager.cc:194] Registered new tserver with Master: 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553)
I20260929 02:49:21.867455 14270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:49:21.896732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33279
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:22.250770 14465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:22.251082 14465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:22.251178 14465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:22.258872 14465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:22.259012 14465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:22.259130 14465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:49:22.269327 14465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:22.270973 14465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:22.272401 14465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:22.287122 14473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:22.287204 14470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:22.287174 14471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:22.287982 14465 server_base.cc:1034] running on GCE node
I20260929 02:49:22.288761 14465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:22.289878 14465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:22.291146 14465 hybrid_clock.cc:648] HybridClock initialized: now 1790650162291072 us; error 56 us; skew 500 ppm
I20260929 02:49:22.291491 14465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:22 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:22.295264 14465 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:49:22.297096 14465 webserver.cc:460] Webserver started at http://127.0.97.2:40717/ using document root <none> and password file <none>
I20260929 02:49:22.297887 14465 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:22.298053 14465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:22.298368 14465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:22.301537 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "e732b4a6622b44a1ae29ba22bf2395e3"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "6b65de3bfd09f26fad067669b66404ad"
server_key_iv: "d0d174083b2b65ee525761617091e88e"
server_key_version: "encryptionkey@0"
I20260929 02:49:22.302644 14465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e732b4a6622b44a1ae29ba22bf2395e3"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "6b65de3bfd09f26fad067669b66404ad"
server_key_iv: "d0d174083b2b65ee525761617091e88e"
server_key_version: "encryptionkey@0"
I20260929 02:49:22.308319 14465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:22.312469 14480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:22.314093 14465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:22.314306 14465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "e732b4a6622b44a1ae29ba22bf2395e3"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "6b65de3bfd09f26fad067669b66404ad"
server_key_iv: "d0d174083b2b65ee525761617091e88e"
server_key_version: "encryptionkey@0"
I20260929 02:49:22.314546 14465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:22.338776 14465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:22.343648 14465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:22.343982 14465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:22.345124 14465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:22.346634 14465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:22.346714 14465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:22.346853 14465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:22.346905 14465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:22.384872 14465 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42817
I20260929 02:49:22.384991 14592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42817 every 8 connection(s)
I20260929 02:49:22.386286 14465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:22.393561 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 14465
I20260929 02:49:22.393754 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:22.394253 388 external_mini_cluster.cc:1428] Setting key 414ff411d723d845872c5c439c4e2e87
Sep 29 02:49:22 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:22.409005 14595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:22.389297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:37715 (local address 127.0.97.62:33279)
0929 02:49:22.389825 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:22.389830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:22.390757 (+ 927us) server_negotiation.cc:407] Connection header received
0929 02:49:22.392730 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:22.392736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:22.392831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:22.392960 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:22.394862 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.395765 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.397512 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.397966 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.398113 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:22.401872 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:22.401883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:22.401886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:22.401926 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:22.405044 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:22.406210 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:22.406215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:22.406217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:22.406383 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:22.406986 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:22.406990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:22.406992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:22.407279 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:22.407423 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:22.408432 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0929 02:49:22.408708 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":299,"threads_started":1}
I20260929 02:49:22.410640 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:22.389826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:37715)
0929 02:49:22.390581 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:22.390626 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:22.392407 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:22.393235 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:22.393250 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:22.393972 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:22.394652 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:22.394673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.395941 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.395945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:22.397337 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:22.397351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.398215 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.398223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:22.398486 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:22.399227 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:22.399252 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:22.401600 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:22.405248 (+ 3648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:22.405261 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:22.405284 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:22.406027 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:22.406523 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:22.406531 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:22.406538 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:22.406819 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:22.407432 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:22.407443 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:22.407964 (+ 521us) client_negotiation.cc:769] Sending connection context
0929 02:49:22.408292 (+ 328us) client_negotiation.cc:240] Negotiation successful
0929 02:49:22.408978 (+ 686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":280,"threads_started":1}
I20260929 02:49:22.412360 14593 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33279
I20260929 02:49:22.412827 14593 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:22.413676 14593 heartbeater.cc:511] Master 127.0.97.62:33279 requested a full tablet report, sending...
I20260929 02:49:22.415498 14270 ts_manager.cc:194] Registered new tserver with Master: e732b4a6622b44a1ae29ba22bf2395e3 (127.0.97.2:42817)
I20260929 02:49:22.416448 14270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:37715
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:49:22.454334 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33279
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:22.809799 14599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:22.810225 14599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:22.810453 14599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:22.819041 14599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:22.819178 14599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:22.819306 14599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:49:22.832660 14599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:22.834515 14599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:22.836406 14599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:22.851658 14605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:22.851691 14604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:22.851816 14607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:22.852530 14599 server_base.cc:1034] running on GCE node
I20260929 02:49:22.853282 14599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:22.854406 14599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:22.855646 14599 hybrid_clock.cc:648] HybridClock initialized: now 1790650162855573 us; error 61 us; skew 500 ppm
I20260929 02:49:22.856032 14599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:22 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:22.859606 14599 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:49:22.861582 14599 webserver.cc:460] Webserver started at http://127.0.97.3:43501/ using document root <none> and password file <none>
I20260929 02:49:22.862411 14599 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:22.862555 14599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:22.862895 14599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:22.866191 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "078edf67d25d485fb25744bfd47758be"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "5c540fbcbec04b7cc5c6e68001897b59"
server_key_iv: "5a28b9fb29c9747c2b82e71f938d0d86"
server_key_version: "encryptionkey@0"
I20260929 02:49:22.867071 14599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "078edf67d25d485fb25744bfd47758be"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "5c540fbcbec04b7cc5c6e68001897b59"
server_key_iv: "5a28b9fb29c9747c2b82e71f938d0d86"
server_key_version: "encryptionkey@0"
I20260929 02:49:22.871797 14459 heartbeater.cc:503] Master 127.0.97.62:33279 was elected leader, sending a full tablet report...
I20260929 02:49:22.873231 14599 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260929 02:49:22.877643 14614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:22.879451 14599 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260929 02:49:22.879699 14599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "078edf67d25d485fb25744bfd47758be"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "5c540fbcbec04b7cc5c6e68001897b59"
server_key_iv: "5a28b9fb29c9747c2b82e71f938d0d86"
server_key_version: "encryptionkey@0"
I20260929 02:49:22.880014 14599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:22.911116 14599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:22.916124 14599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:22.916471 14599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:22.917580 14599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:22.919164 14599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:22.919269 14599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:22.919422 14599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:22.919505 14599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:22.955993 14599 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35019
I20260929 02:49:22.956032 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35019 every 8 connection(s)
I20260929 02:49:22.957556 14599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:22.960433 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 14599
I20260929 02:49:22.960697 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:22.961316 388 external_mini_cluster.cc:1428] Setting key 767e259694ea6156efecccaa2ba35173
Sep 29 02:49:22 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:22.981410 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:22.960814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42873 (local address 127.0.97.62:33279)
0929 02:49:22.961425 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:22.961432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:22.962683 (+ 1251us) server_negotiation.cc:407] Connection header received
0929 02:49:22.965289 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:22.965293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:22.965384 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:22.965485 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:22.967274 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.968118 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.969964 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.970441 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.970594 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:22.974645 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:22.974651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:22.974654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:22.974693 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:22.977853 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:22.978762 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:22.978775 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:22.978778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:22.978916 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:22.979420 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:22.979425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:22.979427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:22.979759 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:22.979924 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:22.980865 (+ 941us) server_negotiation.cc:299] Negotiation successful
0929 02:49:22.981102 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":319,"threads_started":1}
I20260929 02:49:22.982568 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:22.961513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:42873)
0929 02:49:22.962502 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:22.962560 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:22.964858 (+ 2298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:22.965793 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:22.965803 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:22.966400 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:22.967051 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:22.967072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.968309 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.968313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:22.969783 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:22.969797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.970723 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:22.970728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:22.970984 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:22.971871 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:22.971896 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:22.974353 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:22.978094 (+ 3741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:22.978103 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:22.978121 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:22.978611 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:22.979037 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:22.979044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:22.979047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:22.979280 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:22.979927 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:22.979935 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:22.980367 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 02:49:22.980734 (+ 367us) client_negotiation.cc:240] Negotiation successful
0929 02:49:22.981248 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":376,"threads_started":1}
I20260929 02:49:22.984408 14727 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33279
I20260929 02:49:22.984826 14727 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:22.985599 14727 heartbeater.cc:511] Master 127.0.97.62:33279 requested a full tablet report, sending...
I20260929 02:49:22.987144 14270 ts_manager.cc:194] Registered new tserver with Master: 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019)
I20260929 02:49:22.988083 14270 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42873
I20260929 02:49:22.990926 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:23.009475 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:22.995789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.0.97.62:33279)
0929 02:49:22.996086 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:22.996095 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:22.996522 (+ 427us) server_negotiation.cc:407] Connection header received
0929 02:49:22.996762 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:22.996766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:22.996844 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:22.996943 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:22.998440 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:22.999184 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.000709 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.001136 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.001236 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.003637 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:23.003642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:23.003645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:23.003690 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:23.006491 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.007298 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.007302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.007305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.007444 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.007794 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.007797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.007799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.008119 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:23.008292 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.008806 (+ 514us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.009050 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 02:49:23.014304 14270 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:23.021591 14270 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:23.047695 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.035960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.0.97.1:41553)
0929 02:49:23.038155 (+ 2195us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.038163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.038193 (+ 30us) server_negotiation.cc:407] Connection header received
0929 02:49:23.038293 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.038301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.038568 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.039491 (+ 923us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:23.039745 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.041464 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.044992 (+ 3528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.046433 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.046594 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.046867 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.047157 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.047335 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2004,"thread_start_us":1845,"threads_started":1}
I20260929 02:49:23.047808 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.035569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:57686)
0929 02:49:23.036581 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:23.036632 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:23.036856 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:23.038874 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:23.038882 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:23.038921 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:23.039540 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.039558 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.041614 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.041621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.044766 (+ 3145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.044774 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.046645 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.046651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.046863 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.046928 (+ 65us) client_negotiation.cc:769] Sending connection context
0929 02:49:23.047108 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 02:49:23.047373 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":232,"thread_start_us":565,"threads_started":1}
I20260929 02:49:23.048257 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.037609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35019 (local address 127.0.0.1:34640)
0929 02:49:23.039090 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:23.039111 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:23.039227 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:23.040808 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:23.040812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:23.040841 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:23.041196 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.041206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.043078 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.043086 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.044955 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.044971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.047155 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.047162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.047494 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.047607 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 02:49:23.047702 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:49:23.047878 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":92,"thread_start_us":1154,"threads_started":1}
I20260929 02:49:23.048744 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.036383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.97.2:42817)
0929 02:49:23.037732 (+ 1349us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.037747 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.037783 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:49:23.037905 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.037914 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.038184 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.038388 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:23.039888 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.041712 (+ 1824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.045246 (+ 3534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.046670 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.046781 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.046967 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.048171 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.048384 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":1676,"threads_started":1}
I20260929 02:49:23.049080 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.035826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42817 (local address 127.0.0.1:34194)
0929 02:49:23.037445 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:23.037464 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:23.037569 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:23.039061 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:23.039072 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:23.039106 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:23.039649 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.039662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.041869 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.041880 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.044755 (+ 2875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.044773 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.047170 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.047175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.047497 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.047607 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 02:49:23.047702 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:49:23.048702 (+ 1000us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"mutex_wait_us":1118,"thread_start_us":1385,"threads_started":1}
I20260929 02:49:23.049865 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.038205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34640 (local address 127.0.97.3:35019)
0929 02:49:23.040289 (+ 2084us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.040295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.040323 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:49:23.040399 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.040404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.040629 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.040831 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:23.041465 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.042897 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.045138 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.046609 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.046754 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.046977 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.049336 (+ 2359us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.049519 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1910,"thread_start_us":1753,"threads_started":1}
I20260929 02:49:23.052573 14394 tablet_service.cc:1467] Processing CreateTablet for tablet de15d1475b08448c8023de448c751d78 (DEFAULT_TABLE table=test-table [id=99492901438640558a18baf976d129d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:23.052981 14662 tablet_service.cc:1467] Processing CreateTablet for tablet de15d1475b08448c8023de448c751d78 (DEFAULT_TABLE table=test-table [id=99492901438640558a18baf976d129d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:23.053508 14528 tablet_service.cc:1467] Processing CreateTablet for tablet de15d1475b08448c8023de448c751d78 (DEFAULT_TABLE table=test-table [id=99492901438640558a18baf976d129d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:23.054227 14662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de15d1475b08448c8023de448c751d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:23.054242 14394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de15d1475b08448c8023de448c751d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:23.054754 14528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de15d1475b08448c8023de448c751d78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:23.066754 14746 tablet_bootstrap.cc:492] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Bootstrap starting.
I20260929 02:49:23.069744 14747 tablet_bootstrap.cc:492] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Bootstrap starting.
I20260929 02:49:23.071592 14748 tablet_bootstrap.cc:492] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Bootstrap starting.
I20260929 02:49:23.072202 14746 tablet_bootstrap.cc:654] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:23.073339 14747 tablet_bootstrap.cc:654] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:23.073707 14746 log.cc:824] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:23.074630 14747 log.cc:824] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:23.074750 14748 tablet_bootstrap.cc:654] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:23.075785 14748 log.cc:824] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:23.077517 14746 tablet_bootstrap.cc:492] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: No bootstrap required, opened a new log
I20260929 02:49:23.077827 14746 ts_tablet_manager.cc:1397] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 02:49:23.078933 14748 tablet_bootstrap.cc:492] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: No bootstrap required, opened a new log
I20260929 02:49:23.079197 14748 ts_tablet_manager.cc:1397] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 02:49:23.082062 14747 tablet_bootstrap.cc:492] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: No bootstrap required, opened a new log
I20260929 02:49:23.082403 14747 ts_tablet_manager.cc:1397] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260929 02:49:23.089385 14746 raft_consensus.cc:348] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.089963 14746 raft_consensus.cc:374] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:23.090076 14746 raft_consensus.cc:729] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078edf67d25d485fb25744bfd47758be, State: Initialized, Role: FOLLOWER
I20260929 02:49:23.090263 14748 raft_consensus.cc:348] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.090718 14748 raft_consensus.cc:374] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:23.090834 14748 raft_consensus.cc:729] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e732b4a6622b44a1ae29ba22bf2395e3, State: Initialized, Role: FOLLOWER
I20260929 02:49:23.090749 14747 raft_consensus.cc:348] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.090937 14746 consensus_queue.cc:260] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.091207 14747 raft_consensus.cc:374] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:23.091302 14747 raft_consensus.cc:729] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5344d6ef23db4e5a921786826cfd1099, State: Initialized, Role: FOLLOWER
I20260929 02:49:23.091456 14748 consensus_queue.cc:260] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.091903 14747 consensus_queue.cc:260] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.093115 14593 heartbeater.cc:503] Master 127.0.97.62:33279 was elected leader, sending a full tablet report...
I20260929 02:49:23.094252 14748 ts_tablet_manager.cc:1428] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260929 02:49:23.098680 14727 heartbeater.cc:503] Master 127.0.97.62:33279 was elected leader, sending a full tablet report...
I20260929 02:49:23.098853 14747 ts_tablet_manager.cc:1428] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20260929 02:49:23.101293 14746 ts_tablet_manager.cc:1428] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Time spent starting tablet: real 0.023s user 0.014s sys 0.007s
W20260929 02:49:23.138834 14594 tablet.cc:2369] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:49:23.209504 14728 tablet.cc:2369] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:23.247432 14753 raft_consensus.cc:482] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:23.247821 14753 raft_consensus.cc:504] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.249743 14753 leader_election.cc:288] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553), 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019)
I20260929 02:49:23.259886 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.250284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38839 (local address 127.0.97.3:35019)
0929 02:49:23.250541 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.250551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.251529 (+ 978us) server_negotiation.cc:407] Connection header received
0929 02:49:23.251617 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.251622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.251712 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.251800 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:23.252670 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.253825 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.257198 (+ 3373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.258489 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.258647 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.258765 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.259511 (+ 746us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.259648 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 02:49:23.259876 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.250393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:35871 (local address 127.0.97.1:41553)
0929 02:49:23.250787 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.250796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.251137 (+ 341us) server_negotiation.cc:407] Connection header received
0929 02:49:23.251449 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.251495 (+ 46us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.251607 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.251728 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:23.252670 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.253823 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.257198 (+ 3375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.258480 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.258647 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.258765 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.259456 (+ 691us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.259607 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260929 02:49:23.260138 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.250154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35019 (local address 127.0.97.2:38839)
0929 02:49:23.251395 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:23.251410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:23.251523 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:23.251847 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:23.251853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:23.251925 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:23.252425 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.252445 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.254018 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.254023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.256966 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.256988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.258846 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.258855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.259145 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.259272 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 02:49:23.259456 (+ 184us) client_negotiation.cc:240] Negotiation successful
0929 02:49:23.259570 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":302,"spinlock_wait_cycles":896,"thread_start_us":498,"threads_started":1}
I20260929 02:49:23.260499 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.250108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.97.2:35871)
0929 02:49:23.251012 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:23.251029 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:23.251230 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:23.251847 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:23.251853 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:23.251885 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:23.252487 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.252496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.254017 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.254023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.256966 (+ 2943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:23.256988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.258846 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.258855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:23.259147 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.259270 (+ 123us) client_negotiation.cc:769] Sending connection context
0929 02:49:23.259402 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 02:49:23.259570 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":351,"thread_start_us":882,"threads_started":1}
I20260929 02:49:23.261432 14682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de15d1475b08448c8023de448c751d78" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078edf67d25d485fb25744bfd47758be" is_pre_election: true
I20260929 02:49:23.261720 14414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de15d1475b08448c8023de448c751d78" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5344d6ef23db4e5a921786826cfd1099" is_pre_election: true
I20260929 02:49:23.261983 14682 raft_consensus.cc:2454] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 0.
I20260929 02:49:23.262342 14414 raft_consensus.cc:2454] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 0.
I20260929 02:49:23.262887 14484 leader_election.cc:302] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078edf67d25d485fb25744bfd47758be, e732b4a6622b44a1ae29ba22bf2395e3; no voters:
I20260929 02:49:23.263422 14753 raft_consensus.cc:2781] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:23.263587 14753 raft_consensus.cc:482] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:23.263724 14753 raft_consensus.cc:3037] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:23.266131 14753 raft_consensus.cc:504] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.266816 14753 leader_election.cc:288] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 election: Requested vote from peers 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553), 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019)
I20260929 02:49:23.267511 14682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de15d1475b08448c8023de448c751d78" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078edf67d25d485fb25744bfd47758be"
I20260929 02:49:23.267513 14414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de15d1475b08448c8023de448c751d78" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5344d6ef23db4e5a921786826cfd1099"
I20260929 02:49:23.267756 14682 raft_consensus.cc:3037] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:23.267755 14414 raft_consensus.cc:3037] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:23.271162 14682 raft_consensus.cc:2454] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 1.
I20260929 02:49:23.271163 14414 raft_consensus.cc:2454] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 1.
I20260929 02:49:23.271771 14483 leader_election.cc:302] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 078edf67d25d485fb25744bfd47758be, 5344d6ef23db4e5a921786826cfd1099, e732b4a6622b44a1ae29ba22bf2395e3; no voters:
I20260929 02:49:23.272154 14753 raft_consensus.cc:2781] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:23.272747 14753 raft_consensus.cc:686] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 LEADER]: Becoming Leader. State: Replica: e732b4a6622b44a1ae29ba22bf2395e3, State: Running, Role: LEADER
I20260929 02:49:23.273295 14753 consensus_queue.cc:237] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } }
I20260929 02:49:23.278353 14269 catalog_manager.cc:5495] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e732b4a6622b44a1ae29ba22bf2395e3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } health_report { overall_health: UNKNOWN } } }
W20260929 02:49:23.333158 14460 tablet.cc:2369] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:23.348506 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.342838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.0.97.2:42817)
0929 02:49:23.343044 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.343049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.343069 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:49:23.343164 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.343168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.343251 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.343401 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:23.343796 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.344663 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.346182 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.346635 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.346760 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.347269 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:23.347851 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:23.348005 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.348183 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.348272 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:49:23.357729 14682 raft_consensus.cc:1261] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Refusing update from remote peer e732b4a6622b44a1ae29ba22bf2395e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:23.357729 14414 raft_consensus.cc:1261] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Refusing update from remote peer e732b4a6622b44a1ae29ba22bf2395e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:23.359175 14757 consensus_queue.cc:1035] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:23.359606 14753 consensus_queue.cc:1035] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:23.372020 14760 mvcc.cc:205] Tried to move back new op lower bound from 7334503069101166592 to 7334503068771340288. Current Snapshot: MvccSnapshot[applied={T|T < 7334503069101166592}]
I20260929 02:49:23.378304 14761 mvcc.cc:205] Tried to move back new op lower bound from 7334503069101166592 to 7334503068771340288. Current Snapshot: MvccSnapshot[applied={T|T < 7334503069101166592}]
I20260929 02:49:23.401212 14269 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:23.401512 14269 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:23.405012 14269 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Sending DeleteTablet for 3 replicas of tablet de15d1475b08448c8023de448c751d78
I20260929 02:49:23.406265 14394 tablet_service.cc:1514] Processing DeleteTablet for tablet de15d1475b08448c8023de448c751d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57686
I20260929 02:49:23.406390 14662 tablet_service.cc:1514] Processing DeleteTablet for tablet de15d1475b08448c8023de448c751d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34640
I20260929 02:49:23.406472 14528 tablet_service.cc:1514] Processing DeleteTablet for tablet de15d1475b08448c8023de448c751d78 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:23 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34194
I20260929 02:49:23.407992 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 14331
I20260929 02:49:23.421661 14772 tablet_replica.cc:330] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: stopping tablet replica
I20260929 02:49:23.421950 14770 tablet_replica.cc:330] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: stopping tablet replica
I20260929 02:49:23.422181 14772 raft_consensus.cc:2229] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:23.422407 14770 raft_consensus.cc:2229] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:23.422896 14770 raft_consensus.cc:2258] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:23.422899 14772 raft_consensus.cc:2258] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:23.425249 14770 ts_tablet_manager.cc:1905] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:23.430572 14772 ts_tablet_manager.cc:1905] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:23.432840 14770 ts_tablet_manager.cc:1918] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:23.433022 14770 log.cc:1196] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/de15d1475b08448c8023de448c751d78
I20260929 02:49:23.433317 14771 tablet_replica.cc:330] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: stopping tablet replica
I20260929 02:49:23.433633 14770 ts_tablet_manager.cc:1939] T de15d1475b08448c8023de448c751d78 P 5344d6ef23db4e5a921786826cfd1099: Deleting consensus metadata
I20260929 02:49:23.433805 14771 raft_consensus.cc:2229] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:23.434383 14771 raft_consensus.cc:2258] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 29 02:49:23 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:49:23.435743 14256 catalog_manager.cc:4841] TS 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553): tablet de15d1475b08448c8023de448c751d78 (table test-table [id=99492901438640558a18baf976d129d1]) successfully deleted
I20260929 02:49:23.438519 14771 ts_tablet_manager.cc:1905] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:23.443027 14772 ts_tablet_manager.cc:1918] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:23.443253 14772 log.cc:1196] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/de15d1475b08448c8023de448c751d78
I20260929 02:49:23.443871 14772 ts_tablet_manager.cc:1939] T de15d1475b08448c8023de448c751d78 P e732b4a6622b44a1ae29ba22bf2395e3: Deleting consensus metadata
I20260929 02:49:23.445516 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.409610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.0.97.1:41553)
0929 02:49:23.409861 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.409870 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.420264 (+ 10394us) server_negotiation.cc:407] Connection header received
0929 02:49:23.423705 (+ 3441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.423714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.423814 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.423945 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:23.426185 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.427342 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.429452 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.430136 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.430251 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.436819 (+ 6568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:23.436832 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:23.436849 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:23.436895 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:23.441200 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.442300 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.442311 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.442315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.442498 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.443068 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.443074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.443077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.443792 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:23.444191 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.444866 (+ 675us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.445144 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:49:23.445916 14257 catalog_manager.cc:4841] TS e732b4a6622b44a1ae29ba22bf2395e3 (127.0.97.2:42817): tablet de15d1475b08448c8023de448c751d78 (table test-table [id=99492901438640558a18baf976d129d1]) successfully deleted
I20260929 02:49:23.447281 14454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:23.447340 14771 ts_tablet_manager.cc:1918] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:23.447511 14771 log.cc:1196] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/de15d1475b08448c8023de448c751d78
I20260929 02:49:23.448232 14771 ts_tablet_manager.cc:1939] T de15d1475b08448c8023de448c751d78 P 078edf67d25d485fb25744bfd47758be: Deleting consensus metadata
I20260929 02:49:23.450564 14257 catalog_manager.cc:4841] TS 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019): tablet de15d1475b08448c8023de448c751d78 (table test-table [id=99492901438640558a18baf976d129d1]) successfully deleted
I20260929 02:49:23.541743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 14331
I20260929 02:49:23.566236 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 14465
Sep 29 02:49:23 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:49:23.586352 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.567024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.0.97.2:42817)
0929 02:49:23.567235 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.567242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.567265 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:49:23.567403 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.567409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.567494 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.567625 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:23.569184 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.570445 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.571902 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.572613 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.572776 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.577431 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:23.577452 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:23.577474 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:23.577538 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:23.581934 (+ 4396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.582998 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.583006 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.583009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.583177 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.583655 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.583659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.583702 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.584229 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:23.584503 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.585679 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.585941 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:49:23.587282 14588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:23.705796 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 14465
I20260929 02:49:23.727674 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 14599
Sep 29 02:49:23 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:49:23.746006 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:23.728711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.0.97.3:35019)
0929 02:49:23.728950 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:23.728958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:23.729215 (+ 257us) server_negotiation.cc:407] Connection header received
0929 02:49:23.729500 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:23.729507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:23.729613 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:23.729737 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:23.731558 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.732801 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.734251 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:23.734640 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:23.734745 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:23.739909 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:23.739926 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:23.739942 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:23.739985 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:23.743095 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.743941 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.743949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.743951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.744134 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:23.744515 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:23.744518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:23.744520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:23.744961 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:23.745224 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:23.745530 (+ 306us) server_negotiation.cc:299] Negotiation successful
0929 02:49:23.745703 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:49:23.746801 14722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:23.839388 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 14599
I20260929 02:49:23.856314 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 14238
I20260929 02:49:23.857666 14300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:23.973439 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 14238
I20260929 02:49:23.997375 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33279
--webserver_interface=127.0.97.62
--webserver_port=37755
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:24.360605 14778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:24.360944 14778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:24.361111 14778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:24.369895 14778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:24.370046 14778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:24.370100 14778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:24.370194 14778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:24.370265 14778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:24.381234 14778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:24.383068 14778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:24.384613 14778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:24.409507 14787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:24.409530 14785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:24.409479 14784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:24.414625 14778 server_base.cc:1034] running on GCE node
I20260929 02:49:24.415697 14778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:24.417529 14778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:24.418814 14778 hybrid_clock.cc:648] HybridClock initialized: now 1790650164418747 us; error 51 us; skew 500 ppm
I20260929 02:49:24.419281 14778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:24 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:24.423094 14778 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:24.425094 14778 webserver.cc:460] Webserver started at http://127.0.97.62:37755/ using document root <none> and password file <none>
I20260929 02:49:24.425956 14778 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:24.426126 14778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:24.432649 14778 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.007s sys 0.000s
I20260929 02:49:24.436053 14794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:24.437790 14778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:24.438024 14778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "c85173b927e446c9afdd21998d985120"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "7013927c8685e2511d41b8a7c9801bb2"
server_key_iv: "757648138be9290e3bea59a25ebc1d78"
server_key_version: "encryptionkey@0"
I20260929 02:49:24.439273 14778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:24.458245 14778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:24.463704 14778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:24.464144 14778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:24.485277 14778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33279
I20260929 02:49:24.485311 14845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33279 every 8 connection(s)
I20260929 02:49:24.487015 14778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:24.494127 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 14778
I20260929 02:49:24.494961 14846 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:49:24.494969 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41553
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33279
--webserver_port=46665
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:24.499794 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Bootstrap starting.
I20260929 02:49:24.522508 14846 log.cc:824] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:24.535784 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:49:24.536410 14846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Bootstrap complete.
I20260929 02:49:24.546444 14846 raft_consensus.cc:348] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:24.547441 14846 raft_consensus.cc:729] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85173b927e446c9afdd21998d985120, State: Initialized, Role: FOLLOWER
I20260929 02:49:24.548127 14846 consensus_queue.cc:260] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:24.548400 14846 raft_consensus.cc:388] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:24.548588 14846 raft_consensus.cc:482] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:24.548795 14846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:49:24.553153 14846 raft_consensus.cc:504] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:24.553771 14846 leader_election.cc:302] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85173b927e446c9afdd21998d985120; no voters:
I20260929 02:49:24.554643 14846 leader_election.cc:288] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:49:24.554886 14850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:49:24.556347 14850 raft_consensus.cc:686] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 LEADER]: Becoming Leader. State: Replica: c85173b927e446c9afdd21998d985120, State: Running, Role: LEADER
I20260929 02:49:24.557116 14850 consensus_queue.cc:237] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:24.557578 14846 sys_catalog.cc:564] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:24.559828 14852 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c85173b927e446c9afdd21998d985120. Latest consensus state: current_term: 2 leader_uuid: "c85173b927e446c9afdd21998d985120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } } }
I20260929 02:49:24.560210 14852 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:24.560055 14851 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c85173b927e446c9afdd21998d985120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } } }
I20260929 02:49:24.560488 14851 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:24.572340 14857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:24.582170 14857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:24.583766 14857 catalog_manager.cc:1259] Loaded cluster ID: 5eaf707e6c5a466f9902821e77c7ed7a
I20260929 02:49:24.583964 14857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:24.587388 14857 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 02:49:24.606556 14857 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 02:49:24.610006 14867 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 02:49:24.616340 14857 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:24.618238 14857 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Loaded TSK: 0
I20260929 02:49:24.619930 14857 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:49:24.880870 14848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:24.881232 14848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:24.881410 14848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:24.889446 14848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:24.889645 14848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:24.889829 14848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:49:24.900956 14848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46665
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:24.902717 14848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:24.904211 14848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:24.920728 14874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:24.920754 14873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:24.921340 14876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:24.921957 14848 server_base.cc:1034] running on GCE node
I20260929 02:49:24.922645 14848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:24.923738 14848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:24.925053 14848 hybrid_clock.cc:648] HybridClock initialized: now 1790650164924986 us; error 50 us; skew 500 ppm
I20260929 02:49:24.925457 14848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:24 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:24.929168 14848 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:49:24.931017 14848 webserver.cc:460] Webserver started at http://127.0.97.1:46665/ using document root <none> and password file <none>
I20260929 02:49:24.931785 14848 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:24.931952 14848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:24.938210 14848 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:24.941622 14883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:24.943427 14848 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 02:49:24.943629 14848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "5344d6ef23db4e5a921786826cfd1099"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "882ec58f47d320d254ee3639bd8c3dca"
server_key_iv: "f04774914cc5576eb0362df002ab1869"
server_key_version: "encryptionkey@0"
I20260929 02:49:24.944821 14848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:24.968658 14848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:24.973650 14848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:24.973953 14848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:24.975095 14848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:24.976630 14848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:24.976742 14848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:24.976882 14848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:24.976997 14848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:25.011263 14848 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41553
I20260929 02:49:25.011370 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41553 every 8 connection(s)
I20260929 02:49:25.012869 14848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:25.020970 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 14848
I20260929 02:49:25.021732 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42817
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33279
--webserver_port=40717
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:49:25 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:25.041869 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:25.015958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60899 (local address 127.0.97.62:33279)
0929 02:49:25.016708 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:25.016725 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:25.017141 (+ 416us) server_negotiation.cc:407] Connection header received
0929 02:49:25.019201 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:25.019214 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:25.019560 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:25.019856 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:25.021932 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.023213 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.025923 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.026649 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.026825 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:25.031440 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:25.031455 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:25.031483 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:25.031546 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:25.036379 (+ 4833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:25.037753 (+ 1374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:25.037762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:25.037769 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:25.037977 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:25.038573 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:25.038576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:25.038578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:25.039068 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:25.039230 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:25.040310 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0929 02:49:25.040774 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":287,"threads_started":1}
I20260929 02:49:25.042404 14999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:25.016094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:60899)
0929 02:49:25.016955 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:25.017002 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:25.018676 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:25.020028 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:25.020035 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:25.020819 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:25.021646 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:25.021674 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.023413 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.023419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:25.025725 (+ 2306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:25.025745 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.026908 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.026927 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:25.027260 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:25.028081 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:25.028118 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:25.031045 (+ 2927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:25.036681 (+ 5636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:25.036697 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:25.036721 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:25.037566 (+ 845us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:25.038123 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:25.038132 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:25.038139 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:25.038443 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:25.039228 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:25.039241 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:25.039760 (+ 519us) client_negotiation.cc:769] Sending connection context
0929 02:49:25.040105 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 02:49:25.040823 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":380,"threads_started":1}
I20260929 02:49:25.046221 14996 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33279
I20260929 02:49:25.046600 14996 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:25.047465 14996 heartbeater.cc:511] Master 127.0.97.62:33279 requested a full tablet report, sending...
I20260929 02:49:25.050514 14811 ts_manager.cc:194] Registered new tserver with Master: 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553)
I20260929 02:49:25.052870 14811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:60899
W20260929 02:49:25.383412 15000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:25.383721 15000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:25.383826 15000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:25.391602 15000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:25.391718 15000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:25.391835 15000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:49:25.402094 15000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40717
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:25.403363 15000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:25.404755 15000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:25.418756 15005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:25.418780 15008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:25.418756 15006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:25.419842 15000 server_base.cc:1034] running on GCE node
I20260929 02:49:25.420507 15000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:25.421514 15000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:25.422709 15000 hybrid_clock.cc:648] HybridClock initialized: now 1790650165422643 us; error 62 us; skew 500 ppm
I20260929 02:49:25.423044 15000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:25 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:25.426374 15000 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:49:25.427978 15000 webserver.cc:460] Webserver started at http://127.0.97.2:40717/ using document root <none> and password file <none>
I20260929 02:49:25.428959 15000 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:25.429152 15000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:25.435191 15000 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 02:49:25.438308 15015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:25.439852 15000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:49:25.440062 15000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "e732b4a6622b44a1ae29ba22bf2395e3"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "6b65de3bfd09f26fad067669b66404ad"
server_key_iv: "d0d174083b2b65ee525761617091e88e"
server_key_version: "encryptionkey@0"
I20260929 02:49:25.441282 15000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:25.467531 15000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:25.472875 15000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:25.473203 15000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:25.474350 15000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:25.475872 15000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:25.476029 15000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:25.476176 15000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:25.476266 15000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:25.511600 15000 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42817
I20260929 02:49:25.511703 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42817 every 8 connection(s)
I20260929 02:49:25.513274 15000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:25.521883 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15000
I20260929 02:49:25.522626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35019
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33279
--webserver_port=43501
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:49:25 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:25.536387 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:25.516293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38263 (local address 127.0.97.62:33279)
0929 02:49:25.516504 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:25.516511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:25.517878 (+ 1367us) server_negotiation.cc:407] Connection header received
0929 02:49:25.519734 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:25.519739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:25.519826 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:25.519919 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:25.521589 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.522677 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.524586 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.525356 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.525513 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:25.529780 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:25.529786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:25.529790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:25.529835 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:25.532637 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:25.533791 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:25.533795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:25.533797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:25.533931 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:25.534507 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:25.534511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:25.534520 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:25.534799 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:25.534914 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:25.535936 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0929 02:49:25.536143 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:49:25.538319 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:25.516915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:38263)
0929 02:49:25.517690 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:25.517734 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:25.519449 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:25.520135 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:25.520143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:25.520782 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:25.521397 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:25.521415 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.522911 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.522921 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:25.524384 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:25.524399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:25.525614 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:25.525619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:25.525858 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:25.526683 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:25.526710 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:25.529447 (+ 2737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:25.532804 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:25.532833 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:25.532857 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:25.533640 (+ 783us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:25.534060 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:25.534068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:25.534076 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:25.534362 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:25.534938 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:25.534950 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:25.535476 (+ 526us) client_negotiation.cc:769] Sending connection context
0929 02:49:25.535868 (+ 392us) client_negotiation.cc:240] Negotiation successful
0929 02:49:25.536604 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":316,"threads_started":1}
I20260929 02:49:25.540093 15128 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33279
I20260929 02:49:25.540457 15128 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:25.541368 15128 heartbeater.cc:511] Master 127.0.97.62:33279 requested a full tablet report, sending...
I20260929 02:49:25.543138 14811 ts_manager.cc:194] Registered new tserver with Master: e732b4a6622b44a1ae29ba22bf2395e3 (127.0.97.2:42817)
I20260929 02:49:25.544088 14811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38263
W20260929 02:49:25.880427 15131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:25.880779 15131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:25.880939 15131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:25.888495 15131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:25.888638 15131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:25.888820 15131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:49:25.899037 15131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43501
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:25.900406 15131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:25.901858 15131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:25.916970 15137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:25.916998 15136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:25.917788 15131 server_base.cc:1034] running on GCE node
W20260929 02:49:25.917902 15139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:25.918635 15131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:25.919803 15131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:25.921064 15131 hybrid_clock.cc:648] HybridClock initialized: now 1790650165920989 us; error 73 us; skew 500 ppm
I20260929 02:49:25.921422 15131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:25 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:25.925128 15131 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:49:25.926844 15131 webserver.cc:460] Webserver started at http://127.0.97.3:43501/ using document root <none> and password file <none>
I20260929 02:49:25.927631 15131 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:25.927810 15131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:25.934433 15131 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 02:49:25.938050 15146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:25.939652 15131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:49:25.939850 15131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "078edf67d25d485fb25744bfd47758be"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "5c540fbcbec04b7cc5c6e68001897b59"
server_key_iv: "5a28b9fb29c9747c2b82e71f938d0d86"
server_key_version: "encryptionkey@0"
I20260929 02:49:25.940968 15131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:25.954370 15131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:25.959219 15131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:25.959512 15131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:25.960616 15131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:25.962045 15131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:25.962174 15131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:25.962313 15131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:25.962409 15131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:25.998139 15131 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35019
I20260929 02:49:25.998189 15258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35019 every 8 connection(s)
I20260929 02:49:25.999747 15131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:26.000937 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15131
Sep 29 02:49:26 dist-test-slave-6tjh krb5kdc[14225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:26.018808 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.002062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37462 (local address 127.0.97.62:33279)
0929 02:49:26.002262 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.002267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.003355 (+ 1088us) server_negotiation.cc:407] Connection header received
0929 02:49:26.003587 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.003592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.003656 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.003765 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:26.005423 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.006349 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.008071 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.008530 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.008749 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.011606 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:26.011611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:26.011614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:26.011646 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:26.015197 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.016076 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.016083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.016087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.016236 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.016783 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.016788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.016793 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.017084 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:26.017463 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.018067 (+ 604us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.018373 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 02:49:26.024624 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.002825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60725 (local address 127.0.97.62:33279)
0929 02:49:26.003426 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.003432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.004697 (+ 1265us) server_negotiation.cc:407] Connection header received
0929 02:49:26.006709 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.006715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.006791 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.006899 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:26.008594 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.009610 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.011757 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.012616 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.012739 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.017646 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:26.017652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:26.017659 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:26.017697 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:26.021362 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.022280 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.022284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.022287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.022434 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.022914 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.022919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.022922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.023218 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:26.023333 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.024089 (+ 756us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.024264 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":362,"threads_started":1}
I20260929 02:49:26.025688 15263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.003610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:60725)
0929 02:49:26.004499 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.004580 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.006453 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.007165 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.007174 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.007721 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:26.008347 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.008365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.009791 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.009803 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.011563 (+ 1760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.011579 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.013064 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.013072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.013402 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.014129 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:26.014163 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:26.017362 (+ 3199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:26.021555 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:26.021566 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:26.021588 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:26.022143 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:26.022558 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:26.022561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:26.022564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:26.022773 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:26.023360 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:26.023369 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:26.023727 (+ 358us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.024015 (+ 288us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.024443 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":386,"threads_started":1}
I20260929 02:49:26.027237 15259 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33279
I20260929 02:49:26.027521 15259 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:26.028201 15259 heartbeater.cc:511] Master 127.0.97.62:33279 requested a full tablet report, sending...
I20260929 02:49:26.029664 14811 ts_manager.cc:194] Registered new tserver with Master: 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019)
I20260929 02:49:26.030539 14811 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:60725
I20260929 02:49:26.034579 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:26.041703 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.035614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.0.97.62:33279)
0929 02:49:26.035902 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.035909 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.036229 (+ 320us) server_negotiation.cc:407] Connection header received
0929 02:49:26.036485 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.036489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.036706 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.036859 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:26.037307 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.038031 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.039512 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.039953 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.040056 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.040503 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:26.041115 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:26.041225 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.041331 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.041400 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 02:49:26.042651 14811 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:26.050405 14811 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:26.056470 14996 heartbeater.cc:503] Master 127.0.97.62:33279 was elected leader, sending a full tablet report...
I20260929 02:49:26.076105 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.066355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42817 (local address 127.0.0.1:56710)
0929 02:49:26.067233 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.067252 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.067354 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.068534 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.068538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.068601 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:26.068954 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.068963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.070407 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.070415 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.072897 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.072910 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.075053 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.075058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.075310 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.075378 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.075512 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.075728 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"mutex_wait_us":206,"thread_start_us":812,"threads_started":1}
I20260929 02:49:26.076253 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.063362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35019 (local address 127.0.0.1:44672)
0929 02:49:26.065568 (+ 2206us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.065616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.065810 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.067585 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.067597 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.067639 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:26.068001 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.068011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.070220 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.070231 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.073479 (+ 3248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.073491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.075197 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.075203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.075400 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.075460 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.075543 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.075770 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1877,"mutex_wait_us":568,"thread_start_us":2603,"threads_started":1}
I20260929 02:49:26.076958 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.066824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56710 (local address 127.0.97.2:42817)
0929 02:49:26.068011 (+ 1187us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.068021 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.068049 (+ 28us) server_negotiation.cc:407] Connection header received
0929 02:49:26.068139 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.068145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.068359 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.068540 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:26.069131 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.070207 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.073071 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.074648 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.074783 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.075012 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.076401 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.076620 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":899,"threads_started":1}
I20260929 02:49:26.077041 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.063781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.0.97.3:35019)
0929 02:49:26.066462 (+ 2681us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.066474 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.066510 (+ 36us) server_negotiation.cc:407] Connection header received
0929 02:49:26.066608 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.066617 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.066889 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.067101 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:26.068266 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.070022 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.073637 (+ 3615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.074750 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.074874 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.075118 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.076482 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.076711 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2507,"thread_start_us":2357,"threads_started":1}
I20260929 02:49:26.077543 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.065057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.0.97.1:41553)
0929 02:49:26.066800 (+ 1743us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.066809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.066836 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:49:26.067679 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.067689 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.067920 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.068118 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:26.068746 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.070321 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.073306 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.075381 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.075519 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.075766 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.077004 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.077192 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":1911,"threads_started":1}
I20260929 02:49:26.076850 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.064270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50800)
0929 02:49:26.066246 (+ 1976us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.066264 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.067478 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.068117 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.068125 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.068152 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:26.068518 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.068529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.070506 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.070513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.073142 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.073156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.075952 (+ 2796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.075960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.076145 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.076196 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.076291 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.076443 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"mutex_wait_us":30,"thread_start_us":826,"threads_started":1}
I20260929 02:49:26.081326 15063 tablet_service.cc:1467] Processing CreateTablet for tablet 803923c2bede45358785902ca2e413b0 (DEFAULT_TABLE table=test-table [id=663ada3381984fd2878d9fc5ed8387d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:26.081446 15194 tablet_service.cc:1467] Processing CreateTablet for tablet 803923c2bede45358785902ca2e413b0 (DEFAULT_TABLE table=test-table [id=663ada3381984fd2878d9fc5ed8387d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:26.082775 15063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803923c2bede45358785902ca2e413b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:26.082994 15194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803923c2bede45358785902ca2e413b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:26.097404 14931 tablet_service.cc:1467] Processing CreateTablet for tablet 803923c2bede45358785902ca2e413b0 (DEFAULT_TABLE table=test-table [id=663ada3381984fd2878d9fc5ed8387d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:26.099371 14931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 803923c2bede45358785902ca2e413b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:26.099524 15276 tablet_bootstrap.cc:492] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Bootstrap starting.
I20260929 02:49:26.102494 15275 tablet_bootstrap.cc:492] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Bootstrap starting.
I20260929 02:49:26.105329 15276 tablet_bootstrap.cc:654] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:26.106153 15275 tablet_bootstrap.cc:654] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:26.106715 15276 log.cc:824] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:26.107455 15275 log.cc:824] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:26.110176 15276 tablet_bootstrap.cc:492] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: No bootstrap required, opened a new log
I20260929 02:49:26.110468 15276 ts_tablet_manager.cc:1397] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 02:49:26.110949 15275 tablet_bootstrap.cc:492] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: No bootstrap required, opened a new log
I20260929 02:49:26.111225 15275 ts_tablet_manager.cc:1397] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260929 02:49:26.114439 15279 tablet_bootstrap.cc:492] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Bootstrap starting.
I20260929 02:49:26.118678 15279 tablet_bootstrap.cc:654] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:26.119390 15276 raft_consensus.cc:348] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.119779 15276 raft_consensus.cc:374] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:26.119871 15276 raft_consensus.cc:729] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078edf67d25d485fb25744bfd47758be, State: Initialized, Role: FOLLOWER
I20260929 02:49:26.120043 15279 log.cc:824] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:26.120528 15276 consensus_queue.cc:260] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.120859 15275 raft_consensus.cc:348] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.121310 15275 raft_consensus.cc:374] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:26.121434 15275 raft_consensus.cc:729] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e732b4a6622b44a1ae29ba22bf2395e3, State: Initialized, Role: FOLLOWER
I20260929 02:49:26.122520 15276 ts_tablet_manager.cc:1428] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260929 02:49:26.122551 15259 heartbeater.cc:503] Master 127.0.97.62:33279 was elected leader, sending a full tablet report...
I20260929 02:49:26.122963 15275 consensus_queue.cc:260] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.124126 15128 heartbeater.cc:503] Master 127.0.97.62:33279 was elected leader, sending a full tablet report...
I20260929 02:49:26.124298 15279 tablet_bootstrap.cc:492] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: No bootstrap required, opened a new log
I20260929 02:49:26.124634 15279 ts_tablet_manager.cc:1397] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260929 02:49:26.126109 15275 ts_tablet_manager.cc:1428] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260929 02:49:26.133112 15279 raft_consensus.cc:348] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.133476 15279 raft_consensus.cc:374] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:26.133605 15279 raft_consensus.cc:729] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5344d6ef23db4e5a921786826cfd1099, State: Initialized, Role: FOLLOWER
I20260929 02:49:26.134286 15279 consensus_queue.cc:260] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.135986 15279 ts_tablet_manager.cc:1428] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260929 02:49:26.158972 15282 raft_consensus.cc:482] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:26.159344 15282 raft_consensus.cc:504] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.161139 15282 leader_election.cc:288] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019), 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553)
I20260929 02:49:26.167713 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.161544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51231 (local address 127.0.97.3:35019)
0929 02:49:26.161757 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.161767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.162430 (+ 663us) server_negotiation.cc:407] Connection header received
0929 02:49:26.162683 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.162687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.162749 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.162848 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:26.163382 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.164120 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.165942 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.166731 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.166886 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.167007 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.167322 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.167449 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:49:26.167825 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.161674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55653 (local address 127.0.97.1:41553)
0929 02:49:26.161893 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.161900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.162657 (+ 757us) server_negotiation.cc:407] Connection header received
0929 02:49:26.162853 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.162862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.162922 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.163019 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:26.163607 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.164317 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.166103 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.166863 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.167020 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.167126 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.167442 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.167570 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:49:26.167866 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.161518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35019 (local address 127.0.97.2:51231)
0929 02:49:26.162302 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.162319 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.162522 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.162865 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.162869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.162892 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:26.163233 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.163243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.164255 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.164262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.165782 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.165794 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.166950 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.166956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.167157 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.167182 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.167293 (+ 111us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.167426 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":19,"thread_start_us":464,"threads_started":1}
I20260929 02:49:26.168387 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.161728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.97.2:55653)
0929 02:49:26.162541 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.162555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.162704 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.163035 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.163039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.163067 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:26.163448 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.163458 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.164467 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.164473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.165942 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.165954 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.167082 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.167088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.167272 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.167294 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.167394 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.167516 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":24,"thread_start_us":457,"threads_started":1}
I20260929 02:49:26.168385 15283 raft_consensus.cc:482] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:26.168699 15283 raft_consensus.cc:504] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.168987 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803923c2bede45358785902ca2e413b0" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078edf67d25d485fb25744bfd47758be" is_pre_election: true
I20260929 02:49:26.169584 15214 raft_consensus.cc:2454] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 0.
I20260929 02:49:26.169591 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803923c2bede45358785902ca2e413b0" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5344d6ef23db4e5a921786826cfd1099" is_pre_election: true
I20260929 02:49:26.170092 14951 raft_consensus.cc:2454] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 0.
I20260929 02:49:26.170401 15019 leader_election.cc:302] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078edf67d25d485fb25744bfd47758be, e732b4a6622b44a1ae29ba22bf2395e3; no voters:
I20260929 02:49:26.170857 15282 raft_consensus.cc:2781] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:26.170899 15283 leader_election.cc:288] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019), e732b4a6622b44a1ae29ba22bf2395e3 (127.0.97.2:42817)
I20260929 02:49:26.171020 15282 raft_consensus.cc:482] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:26.171157 15282 raft_consensus.cc:3037] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:26.174168 15282 raft_consensus.cc:504] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.175073 15282 leader_election.cc:288] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 election: Requested vote from peers 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019), 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553)
I20260929 02:49:26.175550 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803923c2bede45358785902ca2e413b0" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078edf67d25d485fb25744bfd47758be"
I20260929 02:49:26.175702 14951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803923c2bede45358785902ca2e413b0" candidate_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5344d6ef23db4e5a921786826cfd1099"
I20260929 02:49:26.175796 15214 raft_consensus.cc:3037] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:26.175918 14951 raft_consensus.cc:3037] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:26.179276 15214 raft_consensus.cc:2454] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 1.
I20260929 02:49:26.179492 14951 raft_consensus.cc:2454] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e732b4a6622b44a1ae29ba22bf2395e3 in term 1.
I20260929 02:49:26.179909 15019 leader_election.cc:302] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 078edf67d25d485fb25744bfd47758be, e732b4a6622b44a1ae29ba22bf2395e3; no voters:
I20260929 02:49:26.180285 15282 raft_consensus.cc:2781] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:26.180928 15282 raft_consensus.cc:686] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 LEADER]: Becoming Leader. State: Replica: e732b4a6622b44a1ae29ba22bf2395e3, State: Running, Role: LEADER
I20260929 02:49:26.181852 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.173202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35157 (local address 127.0.97.3:35019)
0929 02:49:26.173414 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.173421 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.173761 (+ 340us) server_negotiation.cc:407] Connection header received
0929 02:49:26.174078 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.174083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.174179 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.174249 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:26.175066 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.176261 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.179327 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.180623 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.180740 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.180865 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.181422 (+ 557us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.181586 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:49:26.181655 15282 consensus_queue.cc:237] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } }
I20260929 02:49:26.182516 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.174158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51675 (local address 127.0.97.2:42817)
0929 02:49:26.174349 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.174356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.174416 (+ 60us) server_negotiation.cc:407] Connection header received
0929 02:49:26.174773 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.174782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.174873 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.174966 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:26.175768 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.176842 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.179637 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.181396 (+ 1759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.181556 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.181646 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.182091 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.182254 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:26.182525 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.172970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:35019 (local address 127.0.97.1:35157)
0929 02:49:26.173670 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.173690 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.173888 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.174458 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.174462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.174490 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:26.174890 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.174904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.176475 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.176480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.179128 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.179145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.181012 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.181020 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.181210 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.181245 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.182100 (+ 855us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.182237 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"mutex_wait_us":72,"thread_start_us":377,"threads_started":1}
I20260929 02:49:26.183220 15214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803923c2bede45358785902ca2e413b0" candidate_uuid: "5344d6ef23db4e5a921786826cfd1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "078edf67d25d485fb25744bfd47758be" is_pre_election: true
I20260929 02:49:26.183459 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.173762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42817 (local address 127.0.97.1:51675)
0929 02:49:26.174326 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.174345 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:26.174631 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:26.175131 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:26.175135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:26.175158 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:26.175579 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.175592 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.177031 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.177038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.179435 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:26.179449 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.181698 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.181705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:26.181886 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.181925 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:49:26.182819 (+ 894us) client_negotiation.cc:240] Negotiation successful
0929 02:49:26.183072 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":41,"thread_start_us":584,"threads_started":1}
I20260929 02:49:26.183573 15214 raft_consensus.cc:2379] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5344d6ef23db4e5a921786826cfd1099 in current term 1: Already voted for candidate e732b4a6622b44a1ae29ba22bf2395e3 in this term.
I20260929 02:49:26.184408 15083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "803923c2bede45358785902ca2e413b0" candidate_uuid: "5344d6ef23db4e5a921786826cfd1099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" is_pre_election: true
I20260929 02:49:26.185401 14887 leader_election.cc:302] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5344d6ef23db4e5a921786826cfd1099; no voters: 078edf67d25d485fb25744bfd47758be, e732b4a6622b44a1ae29ba22bf2395e3
I20260929 02:49:26.185904 15283 raft_consensus.cc:2726] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:49:26.188059 14811 catalog_manager.cc:5495] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e732b4a6622b44a1ae29ba22bf2395e3 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e732b4a6622b44a1ae29ba22bf2395e3" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 } health_report { overall_health: UNKNOWN } } }
I20260929 02:49:26.207531 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.202107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.97.2:42817)
0929 02:49:26.202313 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.202319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.202426 (+ 107us) server_negotiation.cc:407] Connection header received
0929 02:49:26.202680 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.202684 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.202774 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.202927 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:26.203288 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.204021 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.205462 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.205859 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.205986 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.206456 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:26.206938 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:26.207107 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.207167 (+ 60us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.207248 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 02:49:26.215731 15214 raft_consensus.cc:1261] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Refusing update from remote peer e732b4a6622b44a1ae29ba22bf2395e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:26.215746 14951 raft_consensus.cc:1261] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Refusing update from remote peer e732b4a6622b44a1ae29ba22bf2395e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:26.216687 15288 consensus_queue.cc:1035] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5344d6ef23db4e5a921786826cfd1099" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 41553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:26.217089 15282 consensus_queue.cc:1035] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "078edf67d25d485fb25744bfd47758be" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:26.243183 14811 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:26.243470 14811 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:26.247440 14811 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Sending DeleteTablet for 3 replicas of tablet 803923c2bede45358785902ca2e413b0
I20260929 02:49:26.248857 14931 tablet_service.cc:1514] Processing DeleteTablet for tablet 803923c2bede45358785902ca2e413b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50800
I20260929 02:49:26.248865 15063 tablet_service.cc:1514] Processing DeleteTablet for tablet 803923c2bede45358785902ca2e413b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:56710
I20260929 02:49:26.249045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 14848
I20260929 02:49:26.250170 15194 tablet_service.cc:1514] Processing DeleteTablet for tablet 803923c2bede45358785902ca2e413b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:26 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44672
I20260929 02:49:26.259212 15301 tablet_replica.cc:330] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: stopping tablet replica
I20260929 02:49:26.259817 15301 raft_consensus.cc:2229] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:26.260434 15301 raft_consensus.cc:2258] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:26.261327 15302 tablet_replica.cc:330] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: stopping tablet replica
I20260929 02:49:26.261426 15300 tablet_replica.cc:330] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: stopping tablet replica
I20260929 02:49:26.261782 15302 raft_consensus.cc:2229] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:26.261864 15300 raft_consensus.cc:2229] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:26.262259 15302 raft_consensus.cc:2258] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:26.262576 15300 raft_consensus.cc:2258] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:26.264384 15302 ts_tablet_manager.cc:1905] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:26.264396 15301 ts_tablet_manager.cc:1905] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:26.271287 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.249903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.0.97.1:41553)
0929 02:49:26.250078 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.250085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.250148 (+ 63us) server_negotiation.cc:407] Connection header received
0929 02:49:26.250368 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.250374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.250501 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.250629 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:26.252987 (+ 2358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.254560 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.257398 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.258515 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.258723 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.263604 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:26.263619 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:26.263636 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:26.263687 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:26.267755 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.268786 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.268797 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.268800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.268979 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.269442 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.269447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.269450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.269966 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:26.270287 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.270640 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.270873 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:49:26.271699 15302 ts_tablet_manager.cc:1918] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:26.271871 15302 log.cc:1196] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/803923c2bede45358785902ca2e413b0
I20260929 02:49:26.272024 15301 ts_tablet_manager.cc:1918] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:26.272202 15301 log.cc:1196] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/803923c2bede45358785902ca2e413b0
I20260929 02:49:26.272375 14991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:26.272449 15302 ts_tablet_manager.cc:1939] T 803923c2bede45358785902ca2e413b0 P 078edf67d25d485fb25744bfd47758be: Deleting consensus metadata
I20260929 02:49:26.272598 15300 ts_tablet_manager.cc:1905] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:26.274514 14798 catalog_manager.cc:4841] TS 078edf67d25d485fb25744bfd47758be (127.0.97.3:35019): tablet 803923c2bede45358785902ca2e413b0 (table test-table [id=663ada3381984fd2878d9fc5ed8387d4]) successfully deleted
I20260929 02:49:26.282401 15300 ts_tablet_manager.cc:1918] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:26.282644 15300 log.cc:1196] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/803923c2bede45358785902ca2e413b0
I20260929 02:49:26.283278 15300 ts_tablet_manager.cc:1939] T 803923c2bede45358785902ca2e413b0 P e732b4a6622b44a1ae29ba22bf2395e3: Deleting consensus metadata
I20260929 02:49:26.285694 14798 catalog_manager.cc:4841] TS e732b4a6622b44a1ae29ba22bf2395e3 (127.0.97.2:42817): tablet 803923c2bede45358785902ca2e413b0 (table test-table [id=663ada3381984fd2878d9fc5ed8387d4]) successfully deleted
I20260929 02:49:26.370522 15301 ts_tablet_manager.cc:1939] T 803923c2bede45358785902ca2e413b0 P 5344d6ef23db4e5a921786826cfd1099: Deleting consensus metadata
I20260929 02:49:26.371595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 14848
W20260929 02:49:26.395918 14797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:41553 (error 108)
I20260929 02:49:26.396101 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 15000
I20260929 02:49:26.400099 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.399625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50826)
0929 02:49:26.399820 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.399950 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 02:49:26.400470 14797 catalog_manager.cc:4584] TS 5344d6ef23db4e5a921786826cfd1099 (127.0.97.1:41553): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 803923c2bede45358785902ca2e413b0: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
I20260929 02:49:26.413698 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.397028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.0.97.2:42817)
0929 02:49:26.397225 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.397231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.397687 (+ 456us) server_negotiation.cc:407] Connection header received
0929 02:49:26.397763 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.397770 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.397881 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.398000 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:26.399457 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.400644 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.403229 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.403938 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.404077 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.407075 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:26.407089 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:26.407105 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:26.407152 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:26.410614 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.411408 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.411418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.411422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.411593 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.411994 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.412000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.412003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.412430 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:26.412777 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.413108 (+ 331us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.413353 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:49:26.414712 15123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:26.454470 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.453933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50834)
0929 02:49:26.454197 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.454302 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 02:49:26.455538 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.455203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50846)
0929 02:49:26.455335 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.455408 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 02:49:26.516177 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.515602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50858)
0929 02:49:26.515855 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.515982 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260929 02:49:26.517355 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.517001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50862)
0929 02:49:26.517156 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.517247 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 02:49:26.538075 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 15000
I20260929 02:49:26.557926 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 15131
I20260929 02:49:26.572757 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.558747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44674 (local address 127.0.97.3:35019)
0929 02:49:26.559041 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:26.559046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:26.559064 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:26.559144 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:26.559149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:26.559231 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:26.559322 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:26.561002 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.561710 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.563104 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:26.563534 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:26.563641 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:26.566650 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:26.566672 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:26.566688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:26.566730 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:26.569724 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.570540 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.570550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.570556 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.570704 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:26.571096 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:26.571102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:26.571105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:26.571500 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:26.571803 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:26.572147 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 02:49:26.572375 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260929 02:49:26.573464 15254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:26.598222 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.597666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50872)
0929 02:49:26.597910 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.598022 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:49:26.599406 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.598951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50876)
0929 02:49:26.599177 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.599268 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 02:49:26.665442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 15131
I20260929 02:49:26.682173 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 14778
I20260929 02:49:26.683411 14841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:26.799844 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:26.799150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:41553 (local address 127.0.0.1:50882)
0929 02:49:26.799457 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:26.799589 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:41553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260929 02:49:26.799944 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 14778
I20260929 02:49:26.822444 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33279
--webserver_interface=127.0.97.62
--webserver_port=37755
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:27.163383 15308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:27.163702 15308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:27.163867 15308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:27.171352 15308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:27.171482 15308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:27.171569 15308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:27.171666 15308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:27.171774 15308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:27.182163 15308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:27.183655 15308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:27.185210 15308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:27.197903 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.198061 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.198061 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:27.198946 15308 server_base.cc:1034] running on GCE node
I20260929 02:49:27.199880 15308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:27.201524 15308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:27.202751 15308 hybrid_clock.cc:648] HybridClock initialized: now 1790650167202678 us; error 82 us; skew 500 ppm
I20260929 02:49:27.203202 15308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:27 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:27.206573 15308 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:27.208278 15308 webserver.cc:460] Webserver started at http://127.0.97.62:37755/ using document root <none> and password file <none>
I20260929 02:49:27.209164 15308 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:27.209322 15308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:27.215447 15308 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:49:27.219151 15323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.220839 15308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:27.221058 15308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "c85173b927e446c9afdd21998d985120"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "7013927c8685e2511d41b8a7c9801bb2"
server_key_iv: "757648138be9290e3bea59a25ebc1d78"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.222129 15308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:27.253450 15308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:27.258138 15308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:27.258467 15308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:27.278636 15308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33279
I20260929 02:49:27.278646 15374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33279 every 8 connection(s)
I20260929 02:49:27.280334 15308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:27.287030 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15308
I20260929 02:49:27.288079 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:41553
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:33279
--webserver_port=46665
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:27.288460 15375 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:49:27.294212 15375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Bootstrap starting.
I20260929 02:49:27.316812 15375 log.cc:824] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:27.333871 15375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:49:27.334503 15375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120: Bootstrap complete.
I20260929 02:49:27.344090 15375 raft_consensus.cc:348] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:27.345175 15375 raft_consensus.cc:729] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c85173b927e446c9afdd21998d985120, State: Initialized, Role: FOLLOWER
I20260929 02:49:27.345850 15375 consensus_queue.cc:260] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:27.346112 15375 raft_consensus.cc:388] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:27.346247 15375 raft_consensus.cc:482] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:27.346426 15375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:49:27.350598 15375 raft_consensus.cc:504] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:27.351135 15375 leader_election.cc:302] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c85173b927e446c9afdd21998d985120; no voters:
I20260929 02:49:27.351840 15375 leader_election.cc:288] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:49:27.351991 15379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:49:27.353267 15379 raft_consensus.cc:686] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [term 3 LEADER]: Becoming Leader. State: Replica: c85173b927e446c9afdd21998d985120, State: Running, Role: LEADER
I20260929 02:49:27.353914 15379 consensus_queue.cc:237] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } }
I20260929 02:49:27.354265 15375 sys_catalog.cc:564] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:27.356683 15381 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c85173b927e446c9afdd21998d985120. Latest consensus state: current_term: 3 leader_uuid: "c85173b927e446c9afdd21998d985120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } } }
I20260929 02:49:27.357117 15381 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:27.356832 15380 sys_catalog.cc:455] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c85173b927e446c9afdd21998d985120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c85173b927e446c9afdd21998d985120" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33279 } } }
I20260929 02:49:27.357266 15380 sys_catalog.cc:458] T 00000000000000000000000000000000 P c85173b927e446c9afdd21998d985120 [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:27.364125 15386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:27.374553 15386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:27.376266 15386 catalog_manager.cc:1259] Loaded cluster ID: 5eaf707e6c5a466f9902821e77c7ed7a
I20260929 02:49:27.376467 15386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 02:49:27.378714 15386 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff007b039ad google::LogMessage::Fail() at ??:0
@ 0x7ff007b0688a google::LogMessage::SendToLog() at ??:0
@ 0x7ff007b034de google::LogMessage::Flush() at ??:0
@ 0x7ff007b04a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff01d536512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff01d533976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff01d5bb610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff01d5bb439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff00a74f345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff00a75291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff00a752799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff00a72deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff00da076db start_thread at ??:0
@ 0x7ff00699d61f clone at ??:0
W20260929 02:49:27.703684 15377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:27.704026 15377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:27.704186 15377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:27.711771 15377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:27.711905 15377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:27.712112 15377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:49:27.722610 15377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:41553
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=46665
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:27.724457 15377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:27.726529 15377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:27.742394 15416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.742400 15419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:27.742434 15417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:27.743319 15377 server_base.cc:1034] running on GCE node
I20260929 02:49:27.743921 15377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:27.745035 15377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:27.746250 15377 hybrid_clock.cc:648] HybridClock initialized: now 1790650167746177 us; error 63 us; skew 500 ppm
I20260929 02:49:27.746551 15377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:27 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:27.750015 15377 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:49:27.751539 15377 webserver.cc:460] Webserver started at http://127.0.97.1:46665/ using document root <none> and password file <none>
I20260929 02:49:27.752233 15377 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:27.752339 15377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:27.758123 15377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:27.761220 15426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.762923 15377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:27.763149 15377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "5344d6ef23db4e5a921786826cfd1099"
format_stamp: "Formatted at 2026-09-29 02:49:21 on dist-test-slave-6tjh"
server_key: "882ec58f47d320d254ee3639bd8c3dca"
server_key_iv: "f04774914cc5576eb0362df002ab1869"
server_key_version: "encryptionkey@0"
I20260929 02:49:27.764189 15377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:27.792658 15377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:27.797569 15377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:27.797875 15377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:27.798969 15377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:27.800467 15377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:27.800575 15377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.800693 15377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:27.800773 15377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:27.836573 15377 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41553
I20260929 02:49:27.836812 15538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41553 every 8 connection(s)
I20260929 02:49:27.838150 15377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:27.843451 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:27.841512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:46401)
0929 02:49:27.842268 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:27.842486 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
W20260929 02:49:27.844271 15539 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33279: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
I20260929 02:49:27.845321 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:27.844944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:37473)
0929 02:49:27.845108 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:27.845197 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:49:27.846211 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:27.845856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:38665)
0929 02:49:27.846007 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:27.846098 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:49:27.846463 15539 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:49:27.848119 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15377
I20260929 02:49:27.848904 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42817
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:33279
--webserver_port=40717
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:28.195690 15542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:28.196000 15542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:28.196156 15542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:28.203843 15542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:28.203967 15542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:28.204149 15542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:49:28.214715 15542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40717
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:28.216143 15542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:28.217597 15542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:28.232228 15547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:28.232092 15548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:28.232141 15550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:28.233451 15542 server_base.cc:1034] running on GCE node
I20260929 02:49:28.234126 15542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:28.235193 15542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:28.236439 15542 hybrid_clock.cc:648] HybridClock initialized: now 1790650168236372 us; error 66 us; skew 500 ppm
I20260929 02:49:28.236819 15542 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:28 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:28.240347 15542 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:49:28.242189 15542 webserver.cc:460] Webserver started at http://127.0.97.2:40717/ using document root <none> and password file <none>
I20260929 02:49:28.242944 15542 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:28.243115 15542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:28.249233 15542 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 02:49:28.252496 15557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.254184 15542 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:49:28.254348 15542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "e732b4a6622b44a1ae29ba22bf2395e3"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "6b65de3bfd09f26fad067669b66404ad"
server_key_iv: "d0d174083b2b65ee525761617091e88e"
server_key_version: "encryptionkey@0"
I20260929 02:49:28.255368 15542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:28.272396 15542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:28.277390 15542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:28.277711 15542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:28.278784 15542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:28.280242 15542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:28.280378 15542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.280519 15542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:28.280642 15542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 02:49:28.316306 15542 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42817
I20260929 02:49:28.316354 15669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42817 every 8 connection(s)
I20260929 02:49:28.317893 15542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:28.323091 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.321016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:40783)
0929 02:49:28.321838 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.322052 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":330,"threads_started":1}
I20260929 02:49:28.323962 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15542
W20260929 02:49:28.324012 15670 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33279: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
I20260929 02:49:28.324687 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:35019
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:33279
--webserver_port=43501
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:49:28.325136 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.324741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:49927)
0929 02:49:28.324918 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.325005 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 02:49:28.326159 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.325788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:53583)
0929 02:49:28.325948 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.326037 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 02:49:28.326440 15670 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:49:28.668308 15673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:28.668699 15673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:28.668886 15673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:28.676359 15673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:28.676486 15673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:28.676684 15673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:49:28.686818 15673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:35019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=43501
--tserver_master_addrs=127.0.97.62:33279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:28.688237 15673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:28.689572 15673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:28.703662 15678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:28.703778 15679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:28.703742 15681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:28.704008 15673 server_base.cc:1034] running on GCE node
I20260929 02:49:28.705053 15673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:28.706100 15673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:28.707331 15673 hybrid_clock.cc:648] HybridClock initialized: now 1790650168707265 us; error 49 us; skew 500 ppm
I20260929 02:49:28.707711 15673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:28 dist-test-slave-6tjh krb5kdc[14225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:28.711301 15673 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:49:28.712992 15673 webserver.cc:460] Webserver started at http://127.0.97.3:43501/ using document root <none> and password file <none>
I20260929 02:49:28.713696 15673 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:28.713825 15673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:28.719331 15673 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 02:49:28.722595 15688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.724377 15673 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 02:49:28.724619 15673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "078edf67d25d485fb25744bfd47758be"
format_stamp: "Formatted at 2026-09-29 02:49:22 on dist-test-slave-6tjh"
server_key: "5c540fbcbec04b7cc5c6e68001897b59"
server_key_iv: "5a28b9fb29c9747c2b82e71f938d0d86"
server_key_version: "encryptionkey@0"
I20260929 02:49:28.725664 15673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:28.744969 15673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:28.749680 15673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:28.749987 15673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:28.751047 15673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:28.752475 15673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:28.752609 15673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.752714 15673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:28.752804 15673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:28.788120 15673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35019
I20260929 02:49:28.788179 15800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35019 every 8 connection(s)
I20260929 02:49:28.789714 15673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:28.794910 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.792902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:58039)
0929 02:49:28.793705 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.793952 (+ 247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":272,"threads_started":1}
W20260929 02:49:28.795630 15801 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:33279 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:33279: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
I20260929 02:49:28.796607 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.796187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:39249)
0929 02:49:28.796358 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.796443 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:49:28.797488 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.797145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:42707)
0929 02:49:28.797290 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.797371 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 02:49:28.797758 15801 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:49:28.799669 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15673
I20260929 02:49:28.847975 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:28.847109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:50075)
0929 02:49:28.847618 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:28.847737 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":256,"threads_started":1}
I20260929 02:49:29.328042 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:29.327162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:32817)
0929 02:49:29.327652 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:29.327754 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":232,"threads_started":1}
I20260929 02:49:29.799211 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:29.798378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:59199)
0929 02:49:29.798866 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:29.798953 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
I20260929 02:49:29.850178 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:29.849285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:37213)
0929 02:49:29.849882 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:29.849979 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":320,"threads_started":1}
I20260929 02:49:30.330113 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:30.329141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:34923)
0929 02:49:30.329797 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:30.329934 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":261,"threads_started":1}
I20260929 02:49:30.801201 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:30.800220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:46639)
0929 02:49:30.800913 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:30.801018 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":342,"threads_started":1}
I20260929 02:49:30.852188 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:30.851356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:40887)
0929 02:49:30.851899 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:30.851997 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":276,"threads_started":1}
I20260929 02:49:31.332185 15812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.331347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:34339)
0929 02:49:31.331915 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:31.332006 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":321,"threads_started":1}
I20260929 02:49:31.803016 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.802265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:57087)
0929 02:49:31.802744 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:31.802828 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":254,"threads_started":1}
I20260929 02:49:31.854176 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:31.853322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:43855)
0929 02:49:31.853875 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:31.853983 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":280,"threads_started":1}
I20260929 02:49:32.334353 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:32.333419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:44731)
0929 02:49:32.333992 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:32.334104 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":270,"threads_started":1}
I20260929 02:49:32.804905 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:32.804103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:58267)
0929 02:49:32.804584 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:32.804669 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":230,"threads_started":1}
I20260929 02:49:32.856307 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:32.855293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:48447)
0929 02:49:32.855955 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:32.856076 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
I20260929 02:49:33.336149 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:33.335396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:51481)
0929 02:49:33.335876 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:33.335969 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":249,"threads_started":1}
I20260929 02:49:33.806793 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:33.806031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:46545)
0929 02:49:33.806534 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:33.806620 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":243,"threads_started":1}
I20260929 02:49:33.858268 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:33.857435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:60557)
0929 02:49:33.857982 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:33.858089 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":276,"threads_started":1}
I20260929 02:49:34.338188 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:34.337424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:37129)
0929 02:49:34.337910 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:34.338016 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":239,"threads_started":1}
I20260929 02:49:34.808772 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:34.807858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:55377)
0929 02:49:34.808445 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:34.808540 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":336,"threads_started":1}
I20260929 02:49:34.860867 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:34.859775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:50715)
0929 02:49:34.860437 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:34.860610 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":308,"threads_started":1}
I20260929 02:49:35.340258 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.339261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:40043)
0929 02:49:35.339882 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.340015 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":261,"threads_started":1}
I20260929 02:49:35.810701 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.809864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:55761)
0929 02:49:35.810392 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.810475 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
I20260929 02:49:35.862929 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:35.861960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:60285)
0929 02:49:35.862551 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:35.862664 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":300,"threads_started":1}
I20260929 02:49:36.342315 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:36.341363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:49769)
0929 02:49:36.342040 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:36.342129 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":341,"threads_started":1}
I20260929 02:49:36.812621 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:36.811772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:53689)
0929 02:49:36.812283 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:36.812371 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":280,"threads_started":1}
I20260929 02:49:36.865037 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:36.864011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:33353)
0929 02:49:36.864659 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:36.864769 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":255,"threads_started":1}
I20260929 02:49:37.344043 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:37.343277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:46623)
0929 02:49:37.343752 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:37.343832 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":235,"threads_started":1}
I20260929 02:49:37.814574 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:37.813746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:56643)
0929 02:49:37.814280 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:37.814371 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":247,"threads_started":1}
I20260929 02:49:37.867470 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:37.866478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:48113)
0929 02:49:37.867086 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:37.867212 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
I20260929 02:49:38.345870 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.345098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:50639)
0929 02:49:38.345624 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.345703 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":326,"threads_started":1}
I20260929 02:49:38.816700 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.815865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:50123)
0929 02:49:38.816350 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.816438 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":259,"threads_started":1}
I20260929 02:49:38.869443 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:38.868500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:49883)
0929 02:49:38.869174 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:38.869267 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":399,"threads_started":1}
I20260929 02:49:39.347743 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:39.346909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:57955)
0929 02:49:39.347444 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:39.347549 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":280,"threads_started":1}
I20260929 02:49:39.818548 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:39.817757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:49213)
0929 02:49:39.818232 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:39.818328 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260929 02:49:39.871229 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:39.870453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:48661)
0929 02:49:39.870981 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:39.871064 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":304,"threads_started":1}
I20260929 02:49:40.349525 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:40.348845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:57241)
0929 02:49:40.349282 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:40.349360 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":216,"threads_started":1}
I20260929 02:49:40.820477 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:40.819684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:47125)
0929 02:49:40.820212 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:40.820299 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":296,"threads_started":1}
I20260929 02:49:40.873064 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:40.872217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:53085)
0929 02:49:40.872778 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:40.872869 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":245,"threads_started":1}
I20260929 02:49:41.351476 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:41.350650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:33427)
0929 02:49:41.351186 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:41.351279 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":301,"threads_started":1}
I20260929 02:49:41.822397 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:41.821548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:37855)
0929 02:49:41.822106 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:41.822198 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":232,"threads_started":1}
I20260929 02:49:41.874779 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:41.874025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:50619)
0929 02:49:41.874486 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:41.874585 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":221,"threads_started":1}
I20260929 02:49:42.353490 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:42.352703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:55777)
0929 02:49:42.353188 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:42.353282 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":235,"threads_started":1}
I20260929 02:49:42.824168 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:42.823428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:47925)
0929 02:49:42.823879 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:42.823963 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260929 02:49:42.876951 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:42.876016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.1:39305)
0929 02:49:42.876516 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:42.876698 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":256,"threads_started":1}
I20260929 02:49:43.355479 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:43.354706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.2:42651)
0929 02:49:43.355190 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:43.355281 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":263,"threads_started":1}
I20260929 02:49:43.804220 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 15377
I20260929 02:49:43.826706 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:43.825525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33279 (local address 127.0.97.3:34499)
0929 02:49:43.826345 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:43.826494 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":288,"threads_started":1}
I20260929 02:49:43.835604 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:43.805518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50954 (local address 127.0.97.1:41553)
0929 02:49:43.806364 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:43.806389 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:43.806448 (+ 59us) server_negotiation.cc:407] Connection header received
0929 02:49:43.808838 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:43.808851 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:43.809337 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:43.809693 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:43.811120 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:43.812461 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:43.813809 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:43.814582 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:43.814752 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:43.823249 (+ 8497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:43.823262 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:43.823289 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:43.823360 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:43.829496 (+ 6136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:43.831187 (+ 1691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:43.831200 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:43.831221 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:43.831554 (+ 333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:43.832320 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:43.832326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:43.832329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:43.833419 (+ 1090us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:43.833839 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:43.834418 (+ 579us) server_negotiation.cc:299] Negotiation successful
0929 02:49:43.834973 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":446,"threads_started":1}
I20260929 02:49:43.837836 15534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:43.942567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 15377
I20260929 02:49:43.959573 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 15542
I20260929 02:49:43.978382 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:43.960636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.0.97.2:42817)
0929 02:49:43.961203 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:43.961218 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:43.961253 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:49:43.962835 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:43.962844 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:43.963299 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:43.963617 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:43.965211 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:43.966449 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:43.967740 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:43.968374 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:43.968507 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:43.971299 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:43.971306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:43.971323 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:43.971360 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:43.974588 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:43.975374 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:43.975383 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:43.975391 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:43.975582 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:43.976091 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:43.976096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:43.976098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:43.976794 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:43.977065 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:43.977604 (+ 539us) server_negotiation.cc:299] Negotiation successful
0929 02:49:43.978088 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":297,"threads_started":1}
I20260929 02:49:43.980063 15665 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:44.065860 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 15542
I20260929 02:49:44.083832 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 15673
I20260929 02:49:44.103574 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:44.085053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.0.97.3:35019)
0929 02:49:44.085683 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:44.085697 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:44.085738 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:49:44.087341 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:44.087351 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:44.087828 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:44.088216 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:44.089934 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.091148 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.092413 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:44.093220 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:44.093436 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:44.096210 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:44.096219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:44.096236 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:44.096277 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:44.099788 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:44.100687 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:44.100696 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:44.100704 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:44.100894 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:44.101340 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:44.101345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:44.101347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:44.101923 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:44.102271 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:44.102782 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 02:49:44.103242 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":303,"threads_started":1}
I20260929 02:49:44.105067 15796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:44.190586 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 15673
2026-09-29T02:49:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24933 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:49:44 dist-test-slave-6tjh krb5kdc[15859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:49:44 dist-test-slave-6tjh krb5kdc[15859](info): set up 2 sockets
Sep 29 02:49:44 dist-test-slave-6tjh krb5kdc[15859](info): commencing operation
krb5kdc: starting...
W20260929 02:49:45.317194 388 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:49:45 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:49:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:49:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:49:45.572964 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41755
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:45.918803 15872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:45.919186 15872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:45.919363 15872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:45.926908 15872 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:49:45.927060 15872 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:49:45.927156 15872 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:49:45.927255 15872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:49:45.927361 15872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:45.927485 15872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:45.927575 15872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:49:45.927659 15872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:49:45.938089 15872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:45.939491 15872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:45.940915 15872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:45.954739 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:45.954806 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:45.954809 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:45.955513 15872 server_base.cc:1034] running on GCE node
I20260929 02:49:45.956597 15872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:45.958280 15872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:45.959610 15872 hybrid_clock.cc:648] HybridClock initialized: now 1790650185959513 us; error 55 us; skew 500 ppm
I20260929 02:49:45.960228 15872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:45 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:45.963764 15872 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:49:45.965528 15872 webserver.cc:460] Webserver started at http://127.0.97.62:37937/ using document root <none> and password file <none>
I20260929 02:49:45.966342 15872 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:45.966466 15872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:45.966804 15872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:45.969939 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "18fa4366f41f4bf399568697d521ebbf"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-6tjh"
server_key: "999d6bb2a2e2c404b016ecef8a2c3ffa"
server_key_iv: "f5f91fe75bca4fb803ceabe9485abae8"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.970737 15872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "18fa4366f41f4bf399568697d521ebbf"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-6tjh"
server_key: "999d6bb2a2e2c404b016ecef8a2c3ffa"
server_key_iv: "f5f91fe75bca4fb803ceabe9485abae8"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.976240 15872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:49:45.980476 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:45.982018 15872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:45.982244 15872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "18fa4366f41f4bf399568697d521ebbf"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-6tjh"
server_key: "999d6bb2a2e2c404b016ecef8a2c3ffa"
server_key_iv: "f5f91fe75bca4fb803ceabe9485abae8"
server_key_version: "encryptionkey@0"
I20260929 02:49:45.982489 15872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:45.999290 15872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:46.003916 15872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:46.004227 15872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:46.023705 15872 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41755
I20260929 02:49:46.023725 15938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41755 every 8 connection(s)
I20260929 02:49:46.025205 15872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:49:46.027660 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15872
I20260929 02:49:46.027875 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:49:46.028616 388 external_mini_cluster.cc:1428] Setting key b3b7419888c8ee2e9a3cc6c5a00615d0
I20260929 02:49:46.029210 15939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:49:46 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:46.043756 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Bootstrap starting.
I20260929 02:49:46.048233 15939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:46.049638 15939 log.cc:824] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:46.053284 15939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: No bootstrap required, opened a new log
I20260929 02:49:46.053525 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:46.030631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.0.97.62:41755)
0929 02:49:46.031445 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:46.031460 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:46.031509 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:49:46.033234 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:46.033263 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:46.033790 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:46.034266 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:46.035563 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.037628 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.039011 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.040040 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.040250 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:46.044384 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:46.044395 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:46.044417 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:46.044465 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:46.048695 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.049467 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.049477 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.049486 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.049721 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.050148 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.050156 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.050162 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.050816 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:46.051154 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:46.051633 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 02:49:46.052109 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":360,"threads_started":1}
I20260929 02:49:46.061874 15939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:49:46.062263 15939 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:46.062407 15939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fa4366f41f4bf399568697d521ebbf, State: Initialized, Role: FOLLOWER
I20260929 02:49:46.063045 15939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:49:46.063278 15939 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:49:46.063416 15939 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:49:46.063589 15939 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:46.066090 15939 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:49:46.066613 15939 leader_election.cc:302] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18fa4366f41f4bf399568697d521ebbf; no voters:
I20260929 02:49:46.067229 15939 leader_election.cc:288] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:49:46.067399 15944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:46.068614 15944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 LEADER]: Becoming Leader. State: Replica: 18fa4366f41f4bf399568697d521ebbf, State: Running, Role: LEADER
I20260929 02:49:46.069264 15944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:49:46.069536 15939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:49:46.072186 15946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fa4366f41f4bf399568697d521ebbf. Latest consensus state: current_term: 1 leader_uuid: "18fa4366f41f4bf399568697d521ebbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } } }
I20260929 02:49:46.072155 15945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18fa4366f41f4bf399568697d521ebbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } } }
I20260929 02:49:46.072626 15946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:46.072728 15945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: This master's current role is: LEADER
I20260929 02:49:46.076251 15951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:49:46.082326 15951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:49:46.092239 15951 catalog_manager.cc:1347] Generated new cluster ID: 99e89ef34e9a4c3abc4593b657e39db6
I20260929 02:49:46.092409 15951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:49:46.104997 15951 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:49:46.106745 15951 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:49:46.115378 15951 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Generated new TSK 0
I20260929 02:49:46.116380 15951 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:49:46.177233 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41755
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:46.525764 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:46.526062 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:46.526227 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:46.533826 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:46.533946 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:46.534117 15965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:49:46.544193 15965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:46.545573 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:46.547005 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:46.561676 15971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:46.561638 15973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:46.561678 15970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:46.562225 15965 server_base.cc:1034] running on GCE node
I20260929 02:49:46.563086 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:46.564172 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:46.565443 15965 hybrid_clock.cc:648] HybridClock initialized: now 1790650186565382 us; error 50 us; skew 500 ppm
I20260929 02:49:46.565863 15965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:46 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:46.569332 15965 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:49:46.571208 15965 webserver.cc:460] Webserver started at http://127.0.97.1:34249/ using document root <none> and password file <none>
I20260929 02:49:46.571969 15965 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:46.572079 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:46.572348 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:46.575619 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "efb83a766ad747d284490367505cd262"
format_stamp: "Formatted at 2026-09-29 02:49:46 on dist-test-slave-6tjh"
server_key: "7ea1af559df370a9f99f21bd5c7acd93"
server_key_iv: "456146da57ee42fd90257c2e42e7fb5f"
server_key_version: "encryptionkey@0"
I20260929 02:49:46.576421 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "efb83a766ad747d284490367505cd262"
format_stamp: "Formatted at 2026-09-29 02:49:46 on dist-test-slave-6tjh"
server_key: "7ea1af559df370a9f99f21bd5c7acd93"
server_key_iv: "456146da57ee42fd90257c2e42e7fb5f"
server_key_version: "encryptionkey@0"
I20260929 02:49:46.582149 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:49:46.586128 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:46.588141 15965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:49:46.588349 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "efb83a766ad747d284490367505cd262"
format_stamp: "Formatted at 2026-09-29 02:49:46 on dist-test-slave-6tjh"
server_key: "7ea1af559df370a9f99f21bd5c7acd93"
server_key_iv: "456146da57ee42fd90257c2e42e7fb5f"
server_key_version: "encryptionkey@0"
I20260929 02:49:46.588627 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:46.636866 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:46.641726 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:46.642050 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:46.643112 15965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:46.644515 15965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:46.644636 15965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:46.644800 15965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:46.644876 15965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:46.679152 15965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46133
I20260929 02:49:46.679205 16092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46133 every 8 connection(s)
I20260929 02:49:46.680529 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:49:46.690065 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 15965
I20260929 02:49:46.690301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:49:46.690943 388 external_mini_cluster.cc:1428] Setting key 548b857fb7d95a83d3b50b977650e7b9
Sep 29 02:49:46 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:46.702351 16095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:46.683182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40419 (local address 127.0.97.62:41755)
0929 02:49:46.683649 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:46.683654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:46.684639 (+ 985us) server_negotiation.cc:407] Connection header received
0929 02:49:46.686651 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:46.686657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:46.686752 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:46.686873 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:46.688425 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.689209 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.690934 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.691398 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.691544 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:46.695944 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:46.695954 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:46.695958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:46.695991 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:46.698623 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.699619 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.699623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.699625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.699748 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:46.700278 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:46.700285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:46.700290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:46.700695 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:46.700858 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:46.701584 (+ 726us) server_negotiation.cc:299] Negotiation successful
0929 02:49:46.701857 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":243,"threads_started":1}
I20260929 02:49:46.703008 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:46.683773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:40419)
0929 02:49:46.684437 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:46.684500 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:46.686361 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:46.687135 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:46.687143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:46.687678 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:46.688249 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:46.688267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.689390 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.689394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:46.690785 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:46.690797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:46.691661 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:46.691668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:46.691924 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:46.692693 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:46.692717 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:46.695705 (+ 2988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:46.698789 (+ 3084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:46.698801 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:46.698822 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:46.699491 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:46.699862 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:46.699870 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:46.699876 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:46.700146 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:46.700817 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:46.700823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:46.701201 (+ 378us) client_negotiation.cc:769] Sending connection context
0929 02:49:46.701442 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 02:49:46.701833 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":278,"threads_started":1}
I20260929 02:49:46.704684 16093 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41755
I20260929 02:49:46.705010 16093 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:46.705739 16093 heartbeater.cc:511] Master 127.0.97.62:41755 requested a full tablet report, sending...
I20260929 02:49:46.708132 15904 ts_manager.cc:194] Registered new tserver with Master: efb83a766ad747d284490367505cd262 (127.0.97.1:46133)
I20260929 02:49:46.710428 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40419
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:49:46.747941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41755
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:47.083465 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:47.083781 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:47.083952 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:47.091558 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:47.091686 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:47.091887 16099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:49:47.102391 16099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:47.103725 16099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:47.105048 16099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:47.119724 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:47.119683 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:47.119724 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:47.120738 16099 server_base.cc:1034] running on GCE node
I20260929 02:49:47.121554 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:47.122690 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:47.123939 16099 hybrid_clock.cc:648] HybridClock initialized: now 1790650187123928 us; error 81 us; skew 500 ppm
I20260929 02:49:47.124338 16099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:47 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:47.127725 16099 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:49:47.129412 16099 webserver.cc:460] Webserver started at http://127.0.97.2:36559/ using document root <none> and password file <none>
I20260929 02:49:47.130149 16099 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:47.130255 16099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:47.130594 16099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:47.133631 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "88a23a0e60446031772dd0a9b9dbc963"
server_key_iv: "893da1cfc2cb5f4b74d2e36e1b3ea384"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.134393 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "88a23a0e60446031772dd0a9b9dbc963"
server_key_iv: "893da1cfc2cb5f4b74d2e36e1b3ea384"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.139500 16099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:49:47.143167 16114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:47.144488 16099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:49:47.144737 16099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "88a23a0e60446031772dd0a9b9dbc963"
server_key_iv: "893da1cfc2cb5f4b74d2e36e1b3ea384"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.144912 16099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:47.175576 16099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:47.180090 16099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:47.180408 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:47.181535 16099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:47.183018 16099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:47.183125 16099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:47.183277 16099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:47.183359 16099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:47.218093 16099 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33645
I20260929 02:49:47.218138 16226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33645 every 8 connection(s)
I20260929 02:49:47.219450 16099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:49:47.222754 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 16099
I20260929 02:49:47.222977 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:49:47.223426 388 external_mini_cluster.cc:1428] Setting key a28810244a6e4a1b5d07fa8393f1e349
Sep 29 02:49:47 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:47.242285 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.222047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38899 (local address 127.0.97.62:41755)
0929 02:49:47.222532 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.222541 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.223611 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 02:49:47.225703 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.225709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.225791 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.225909 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:47.227448 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.228682 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.230725 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.231498 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.231647 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.235945 (+ 4298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:47.235951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:47.235954 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:47.235993 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:47.238701 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.239765 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.239770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.239772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.239902 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.240412 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.240415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.240417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.240757 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:47.240875 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.241829 (+ 954us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.242011 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":237,"threads_started":1}
I20260929 02:49:47.243976 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.222508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:38899)
0929 02:49:47.223345 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:47.223391 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:47.225397 (+ 2006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:47.226122 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:47.226130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:47.226678 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:47.227270 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.227288 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.228833 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.228837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.230554 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.230571 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.231731 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.231739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.232062 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.232936 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:47.232972 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:47.235669 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:47.238884 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:47.238898 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:47.238920 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:47.239631 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:47.240021 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:47.240029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:47.240035 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:47.240304 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:47.240889 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:47.240901 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:47.241410 (+ 509us) client_negotiation.cc:769] Sending connection context
0929 02:49:47.241757 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 02:49:47.242385 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":367,"threads_started":1}
I20260929 02:49:47.245724 16227 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41755
I20260929 02:49:47.246094 16227 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:47.247040 16227 heartbeater.cc:511] Master 127.0.97.62:41755 requested a full tablet report, sending...
I20260929 02:49:47.248834 15904 ts_manager.cc:194] Registered new tserver with Master: fc2e3d84f7d549b1851785d6c4dfbb0a (127.0.97.2:33645)
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:49:47.249848 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:49:47.281092 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41755
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:49:47.622273 16233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:49:47.622602 16233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:49:47.622802 16233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:49:47.630496 16233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:49:47.630643 16233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:49:47.630861 16233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:49:47.641285 16233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:49:47.642822 16233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:49:47.644254 16233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:49:47.658720 16238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:47.658689 16239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:49:47.658650 16241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:49:47.659669 16233 server_base.cc:1034] running on GCE node
I20260929 02:49:47.660342 16233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:49:47.661461 16233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:49:47.662714 16233 hybrid_clock.cc:648] HybridClock initialized: now 1790650187662639 us; error 67 us; skew 500 ppm
I20260929 02:49:47.663082 16233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:49:47 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:49:47.666791 16233 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:49:47.668450 16233 webserver.cc:460] Webserver started at http://127.0.97.3:38473/ using document root <none> and password file <none>
I20260929 02:49:47.669196 16233 fs_manager.cc:359] Metadata directory not provided
I20260929 02:49:47.669303 16233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:49:47.669664 16233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:49:47.672844 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "4ebdcb76cb574884962e675c91800de6"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "3fcb7d65ce6aec6a33ab90c49a623702"
server_key_iv: "2224863ef9e50b17265996344a7f0acf"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.673581 16233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4ebdcb76cb574884962e675c91800de6"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "3fcb7d65ce6aec6a33ab90c49a623702"
server_key_iv: "2224863ef9e50b17265996344a7f0acf"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.679320 16233 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:49:47.683991 16248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:47.685564 16233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:49:47.685714 16233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "4ebdcb76cb574884962e675c91800de6"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "3fcb7d65ce6aec6a33ab90c49a623702"
server_key_iv: "2224863ef9e50b17265996344a7f0acf"
server_key_version: "encryptionkey@0"
I20260929 02:49:47.685883 16233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:49:47.713910 16093 heartbeater.cc:503] Master 127.0.97.62:41755 was elected leader, sending a full tablet report...
I20260929 02:49:47.725724 16233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:49:47.731760 16233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:49:47.732077 16233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:49:47.733222 16233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:49:47.734573 16233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:49:47.734652 16233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:47.734814 16233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:49:47.734898 16233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:49:47.768198 16233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42419
I20260929 02:49:47.768289 16360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42419 every 8 connection(s)
I20260929 02:49:47.769678 16233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:49:47.775939 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 16233
I20260929 02:49:47.776165 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:49:47.776918 388 external_mini_cluster.cc:1428] Setting key 15e1574fe440c6401981baeeb0481d28
Sep 29 02:49:47 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:49:47.793373 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.772508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51351 (local address 127.0.97.62:41755)
0929 02:49:47.773095 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.773100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.774167 (+ 1067us) server_negotiation.cc:407] Connection header received
0929 02:49:47.776433 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.776440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.776628 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.776780 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:47.778602 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.779740 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.781806 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.782236 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.782384 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.786328 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:47.786333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:47.786337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:47.786375 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:47.789377 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.790281 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.790289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.790293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.790484 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.790971 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.790977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.790980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.791380 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:47.791575 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.792581 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.792861 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":286,"threads_started":1}
I20260929 02:49:47.794500 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.773151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:51351)
0929 02:49:47.773970 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:47.774017 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:47.775990 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:47.776997 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:47.777011 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:47.777712 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:49:47.778403 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.778432 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.780045 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.780053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.781629 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.781644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.782489 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.782497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.782768 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.783515 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:49:47.783548 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:49:47.786063 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:49:47.789574 (+ 3511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:47.789605 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:47.789621 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:47.790126 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:47.790607 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:49:47.790613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:49:47.790617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:49:47.790836 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:49:47.791546 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:49:47.791556 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:49:47.791992 (+ 436us) client_negotiation.cc:769] Sending connection context
0929 02:49:47.792434 (+ 442us) client_negotiation.cc:240] Negotiation successful
0929 02:49:47.793194 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":325,"threads_started":1}
I20260929 02:49:47.796257 16361 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41755
I20260929 02:49:47.796703 16361 heartbeater.cc:464] Registering TS with master...
I20260929 02:49:47.797504 16361 heartbeater.cc:511] Master 127.0.97.62:41755 requested a full tablet report, sending...
I20260929 02:49:47.799031 15904 ts_manager.cc:194] Registered new tserver with Master: 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419)
I20260929 02:49:47.799883 15904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:51351
I20260929 02:49:47.806329 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:49:47.826818 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.812574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55632 (local address 127.0.97.62:41755)
0929 02:49:47.812774 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.812779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.813225 (+ 446us) server_negotiation.cc:407] Connection header received
0929 02:49:47.813385 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.813390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.813465 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.813567 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:47.815075 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.816076 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.817815 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.818293 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.818424 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.821093 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:47.821100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:47.821104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:47.821145 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:47.823997 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.824930 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.824936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.824938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.825091 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:47.825510 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:47.825514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:47.825516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:47.825881 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:47.826124 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.826392 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.826591 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:49:47.831077 15904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:49:47.838064 15904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:49:47.862361 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.852123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33645 (local address 127.0.0.1:45904)
0929 02:49:47.852940 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:47.852979 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:47.853121 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:47.854184 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:47.854405 (+ 221us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:47.854455 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:47.854988 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.855001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.856929 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.856937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.859426 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.859445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.861277 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.861283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.861482 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.861541 (+ 59us) client_negotiation.cc:769] Sending connection context
0929 02:49:47.861667 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 02:49:47.861899 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":32,"thread_start_us":320,"threads_started":1}
I20260929 02:49:47.863538 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.852306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.0.97.2:33645)
0929 02:49:47.853314 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.853323 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.853352 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:49:47.853465 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.853474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.853755 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.853952 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:47.855201 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.856784 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.859619 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.861085 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.861242 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.861462 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.862980 (+ 1518us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.863201 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":312,"threads_started":1}
I20260929 02:49:47.863799 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.853932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42419 (local address 127.0.0.1:52018)
0929 02:49:47.854570 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:47.854588 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:47.854715 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:47.856155 (+ 1440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:47.856159 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:47.856193 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:47.856787 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.856808 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.858776 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.858780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.860655 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.860668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.862870 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.862880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.863053 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.863081 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:49:47.863171 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 02:49:47.863322 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":55,"spinlock_wait_cycles":1408,"thread_start_us":825,"threads_started":1}
I20260929 02:49:47.863854 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.853351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46133 (local address 127.0.0.1:33360)
0929 02:49:47.853878 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:47.853898 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:47.854018 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:47.856152 (+ 2134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:47.856157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:47.856198 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:47.856857 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.856869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.858390 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.858394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.860688 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.860702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.862779 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.862786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.862991 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.863111 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 02:49:47.863190 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 02:49:47.863327 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":154,"spinlock_wait_cycles":3840,"thread_start_us":248,"threads_started":1}
I20260929 02:49:47.864710 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.854530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.0.97.3:42419)
0929 02:49:47.855617 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.855625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.855654 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:49:47.855745 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.855751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.855982 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.856161 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:47.857016 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.858610 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.860812 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.862364 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.862494 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.862730 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.864154 (+ 1424us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.864320 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925,"thread_start_us":1264,"threads_started":1}
I20260929 02:49:47.865108 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.853770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33360 (local address 127.0.97.1:46133)
0929 02:49:47.855295 (+ 1525us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.855305 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.855332 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:49:47.855422 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.855429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.855678 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.855871 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:47.857066 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.858233 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.860868 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.862370 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.863060 (+ 690us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.863299 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.863380 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.863563 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1347,"thread_start_us":1192,"threads_started":1}
I20260929 02:49:47.867883 16162 tablet_service.cc:1467] Processing CreateTablet for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e (DEFAULT_TABLE table=test-table [id=fe8110922d7643509da9420f1c6d2cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:47.868175 16028 tablet_service.cc:1467] Processing CreateTablet for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e (DEFAULT_TABLE table=test-table [id=fe8110922d7643509da9420f1c6d2cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:47.868294 16296 tablet_service.cc:1467] Processing CreateTablet for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e (DEFAULT_TABLE table=test-table [id=fe8110922d7643509da9420f1c6d2cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:49:47.869529 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:47.869541 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:47.869660 16296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:49:47.881793 16380 tablet_bootstrap.cc:492] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Bootstrap starting.
I20260929 02:49:47.884335 16381 tablet_bootstrap.cc:492] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Bootstrap starting.
I20260929 02:49:47.885967 16380 tablet_bootstrap.cc:654] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:47.886018 16382 tablet_bootstrap.cc:492] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Bootstrap starting.
I20260929 02:49:47.887326 16380 log.cc:824] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:47.889698 16382 tablet_bootstrap.cc:654] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:47.890340 16380 tablet_bootstrap.cc:492] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: No bootstrap required, opened a new log
I20260929 02:49:47.890623 16380 ts_tablet_manager.cc:1397] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 02:49:47.891063 16382 log.cc:824] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:47.891660 16381 tablet_bootstrap.cc:654] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Neither blocks nor log segments found. Creating new log.
I20260929 02:49:47.893249 16381 log.cc:824] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:49:47.898844 16380 raft_consensus.cc:348] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.899339 16380 raft_consensus.cc:374] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:47.899502 16380 raft_consensus.cc:729] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb83a766ad747d284490367505cd262, State: Initialized, Role: FOLLOWER
I20260929 02:49:47.900183 16380 consensus_queue.cc:260] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.900607 16382 tablet_bootstrap.cc:492] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: No bootstrap required, opened a new log
I20260929 02:49:47.900776 16381 tablet_bootstrap.cc:492] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: No bootstrap required, opened a new log
I20260929 02:49:47.900938 16382 ts_tablet_manager.cc:1397] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260929 02:49:47.901103 16381 ts_tablet_manager.cc:1397] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260929 02:49:47.902057 16380 ts_tablet_manager.cc:1428] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260929 02:49:47.910058 16381 raft_consensus.cc:348] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.910459 16381 raft_consensus.cc:374] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:47.910588 16381 raft_consensus.cc:729] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2e3d84f7d549b1851785d6c4dfbb0a, State: Initialized, Role: FOLLOWER
I20260929 02:49:47.910948 16382 raft_consensus.cc:348] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.911309 16382 raft_consensus.cc:374] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:49:47.911418 16382 raft_consensus.cc:729] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebdcb76cb574884962e675c91800de6, State: Initialized, Role: FOLLOWER
I20260929 02:49:47.911217 16381 consensus_queue.cc:260] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.912010 16382 consensus_queue.cc:260] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.912459 16227 heartbeater.cc:503] Master 127.0.97.62:41755 was elected leader, sending a full tablet report...
I20260929 02:49:47.913021 16381 ts_tablet_manager.cc:1428] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260929 02:49:47.914592 16361 heartbeater.cc:503] Master 127.0.97.62:41755 was elected leader, sending a full tablet report...
I20260929 02:49:47.915149 16382 ts_tablet_manager.cc:1428] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Time spent starting tablet: real 0.014s user 0.010s sys 0.001s
W20260929 02:49:47.933225 16094 tablet.cc:2369] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:47.967320 16387 raft_consensus.cc:482] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:49:47.967716 16387 raft_consensus.cc:504] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.969825 16387 leader_election.cc:288] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efb83a766ad747d284490367505cd262 (127.0.97.1:46133), 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419)
W20260929 02:49:47.971699 16228 tablet.cc:2369] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:47.978291 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.970241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46551 (local address 127.0.97.1:46133)
0929 02:49:47.970526 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.970536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.971484 (+ 948us) server_negotiation.cc:407] Connection header received
0929 02:49:47.971581 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.971586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.971679 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.971788 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:47.972788 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.973731 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.975948 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.977075 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.977204 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.977290 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.977841 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.977993 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:49:47.978263 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.971509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49433 (local address 127.0.97.3:42419)
0929 02:49:47.971719 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:47.971727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:47.972308 (+ 581us) server_negotiation.cc:407] Connection header received
0929 02:49:47.972534 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:47.972591 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:47.972682 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:47.972778 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:49:47.973330 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.974123 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.976113 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.977262 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.977415 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.977528 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:47.977893 (+ 365us) server_negotiation.cc:299] Negotiation successful
0929 02:49:47.978030 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:49:47.978405 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.970258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46133 (local address 127.0.97.2:46551)
0929 02:49:47.971098 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:47.971124 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:47.971302 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:47.972017 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:47.972023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:47.972054 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:47.972578 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.972597 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.973901 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.973904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.975758 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.975772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.977331 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.977341 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.977576 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.977624 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:49:47.977776 (+ 152us) client_negotiation.cc:240] Negotiation successful
0929 02:49:47.977924 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":67,"thread_start_us":452,"threads_started":1}
I20260929 02:49:47.978981 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:47.971157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42419 (local address 127.0.97.2:49433)
0929 02:49:47.972183 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0929 02:49:47.972200 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:49:47.972380 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:49:47.972794 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:49:47.972799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:49:47.972825 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:49:47.973172 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.973180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.974274 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.974280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.975938 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:49:47.975953 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:49:47.977518 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:49:47.977523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:49:47.977669 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:47.977692 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 02:49:47.978554 (+ 862us) client_negotiation.cc:240] Negotiation successful
0929 02:49:47.978720 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":634,"threads_started":1}
I20260929 02:49:47.979954 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b205cccc7c4c0f8b3bd109f69f8c7e" candidate_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb83a766ad747d284490367505cd262" is_pre_election: true
I20260929 02:49:47.980165 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b205cccc7c4c0f8b3bd109f69f8c7e" candidate_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebdcb76cb574884962e675c91800de6" is_pre_election: true
I20260929 02:49:47.980502 16048 raft_consensus.cc:2454] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2e3d84f7d549b1851785d6c4dfbb0a in term 0.
I20260929 02:49:47.980779 16316 raft_consensus.cc:2454] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc2e3d84f7d549b1851785d6c4dfbb0a in term 0.
I20260929 02:49:47.981511 16115 leader_election.cc:302] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efb83a766ad747d284490367505cd262, fc2e3d84f7d549b1851785d6c4dfbb0a; no voters:
I20260929 02:49:47.982056 16387 raft_consensus.cc:2781] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:49:47.982266 16387 raft_consensus.cc:482] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:49:47.982376 16387 raft_consensus.cc:3037] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:47.984752 16387 raft_consensus.cc:504] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.985363 16387 leader_election.cc:288] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [CANDIDATE]: Term 1 election: Requested vote from peers efb83a766ad747d284490367505cd262 (127.0.97.1:46133), 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419)
I20260929 02:49:47.986171 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b205cccc7c4c0f8b3bd109f69f8c7e" candidate_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb83a766ad747d284490367505cd262"
I20260929 02:49:47.986320 16316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b205cccc7c4c0f8b3bd109f69f8c7e" candidate_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebdcb76cb574884962e675c91800de6"
I20260929 02:49:47.986457 16048 raft_consensus.cc:3037] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:47.986590 16316 raft_consensus.cc:3037] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:49:47.989140 16316 raft_consensus.cc:2454] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2e3d84f7d549b1851785d6c4dfbb0a in term 1.
I20260929 02:49:47.989216 16048 raft_consensus.cc:2454] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc2e3d84f7d549b1851785d6c4dfbb0a in term 1.
I20260929 02:49:47.989711 16115 leader_election.cc:302] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efb83a766ad747d284490367505cd262, fc2e3d84f7d549b1851785d6c4dfbb0a; no voters:
I20260929 02:49:47.990062 16387 raft_consensus.cc:2781] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:49:47.990562 16387 raft_consensus.cc:686] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 LEADER]: Becoming Leader. State: Replica: fc2e3d84f7d549b1851785d6c4dfbb0a, State: Running, Role: LEADER
I20260929 02:49:47.991087 16387 consensus_queue.cc:237] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:49:47.996805 15904 catalog_manager.cc:5495] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a reported cstate change: term changed from 0 to 1, leader changed from <none> to fc2e3d84f7d549b1851785d6c4dfbb0a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } health_report { overall_health: UNKNOWN } } }
W20260929 02:49:48.021410 16362 tablet.cc:2369] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:49:48.023237 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.017280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.0.97.2:33645)
0929 02:49:48.017529 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:48.017534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:48.017552 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:49:48.017743 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:48.017746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:48.017824 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:48.017951 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:49:48.018370 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.019126 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.020649 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.021224 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.021340 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:48.022112 (+ 772us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:49:48.022618 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:49:48.022733 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:48.022855 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 02:49:48.022925 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 02:49:48.032079 16316 raft_consensus.cc:1261] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Refusing update from remote peer fc2e3d84f7d549b1851785d6c4dfbb0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:48.032222 16048 raft_consensus.cc:1261] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 1 FOLLOWER]: Refusing update from remote peer fc2e3d84f7d549b1851785d6c4dfbb0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:49:48.033282 16391 consensus_queue.cc:1035] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:48.033699 16387 consensus_queue.cc:1035] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:49:48.050180 16395 mvcc.cc:205] Tried to move back new op lower bound from 7334503170168061952 to 7334503170015813632. Current Snapshot: MvccSnapshot[applied={T|T < 7334503170168061952}]
I20260929 02:49:48.061808 15903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:48.062090 15903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:49:48.065963 15903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Sending DeleteTablet for 3 replicas of tablet f0b205cccc7c4c0f8b3bd109f69f8c7e
I20260929 02:49:48.067207 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45904
I20260929 02:49:48.067198 16028 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33360
I20260929 02:49:48.067684 16296 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b205cccc7c4c0f8b3bd109f69f8c7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:49:48 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:52018
I20260929 02:49:48.068595 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 15965
I20260929 02:49:48.078900 16405 tablet_replica.cc:330] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: stopping tablet replica
I20260929 02:49:48.079121 16404 tablet_replica.cc:330] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: stopping tablet replica
I20260929 02:49:48.079622 16404 raft_consensus.cc:2229] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:48.079660 16405 raft_consensus.cc:2229] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:49:48.080176 16404 raft_consensus.cc:2258] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:48.080379 16405 raft_consensus.cc:2258] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:48.081532 16403 tablet_replica.cc:330] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: stopping tablet replica
I20260929 02:49:48.082052 16403 raft_consensus.cc:2229] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:49:48.082788 16404 ts_tablet_manager.cc:1905] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:48.082787 16405 ts_tablet_manager.cc:1905] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:48.082783 16403 raft_consensus.cc:2258] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:49:48.084926 16403 ts_tablet_manager.cc:1905] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:49:48.090269 16405 ts_tablet_manager.cc:1918] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:48.090607 16405 log.cc:1196] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/f0b205cccc7c4c0f8b3bd109f69f8c7e
I20260929 02:49:48.091188 16405 ts_tablet_manager.cc:1939] T f0b205cccc7c4c0f8b3bd109f69f8c7e P 4ebdcb76cb574884962e675c91800de6: Deleting consensus metadata
I20260929 02:49:48.093416 15888 catalog_manager.cc:4841] TS 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419): tablet f0b205cccc7c4c0f8b3bd109f69f8c7e (table test-table [id=fe8110922d7643509da9420f1c6d2cac]) successfully deleted
I20260929 02:49:48.093943 16403 ts_tablet_manager.cc:1918] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:48.094161 16403 log.cc:1196] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/f0b205cccc7c4c0f8b3bd109f69f8c7e
I20260929 02:49:48.094753 16403 ts_tablet_manager.cc:1939] T f0b205cccc7c4c0f8b3bd109f69f8c7e P fc2e3d84f7d549b1851785d6c4dfbb0a: Deleting consensus metadata
I20260929 02:49:48.095510 16404 ts_tablet_manager.cc:1918] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:49:48.095707 16404 log.cc:1196] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/f0b205cccc7c4c0f8b3bd109f69f8c7e
I20260929 02:49:48.096160 16404 ts_tablet_manager.cc:1939] T f0b205cccc7c4c0f8b3bd109f69f8c7e P efb83a766ad747d284490367505cd262: Deleting consensus metadata
I20260929 02:49:48.096715 15888 catalog_manager.cc:4841] TS fc2e3d84f7d549b1851785d6c4dfbb0a (127.0.97.2:33645): tablet f0b205cccc7c4c0f8b3bd109f69f8c7e (table test-table [id=fe8110922d7643509da9420f1c6d2cac]) successfully deleted
I20260929 02:49:48.097848 15888 catalog_manager.cc:4841] TS efb83a766ad747d284490367505cd262 (127.0.97.1:46133): tablet f0b205cccc7c4c0f8b3bd109f69f8c7e (table test-table [id=fe8110922d7643509da9420f1c6d2cac]) successfully deleted
Sep 29 02:49:48 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:49:48.107223 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.069752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33372 (local address 127.0.97.1:46133)
0929 02:49:48.070012 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:48.070022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:48.077483 (+ 7461us) server_negotiation.cc:407] Connection header received
0929 02:49:48.078749 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:48.078757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:48.078884 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:48.078981 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:48.086671 (+ 7690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.087862 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.092349 (+ 4487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.093014 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.093132 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:48.100935 (+ 7803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:48.100949 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:48.100964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:48.101006 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:48.104007 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.104968 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.104976 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.104978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.105125 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.105542 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.105549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.105556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.105926 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:48.106267 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:48.106587 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 02:49:48.106837 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:49:48.107867 16088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:48.201292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 15965
I20260929 02:49:48.220116 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 16099
Sep 29 02:49:48 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:49:48.242233 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.221955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.0.97.2:33645)
0929 02:49:48.222156 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:48.222162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:48.222183 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:49:48.222250 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:48.222260 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:48.222340 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:48.222469 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:48.224110 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.225453 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.227513 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.228258 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.228399 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:48.234314 (+ 5915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:48.234332 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:48.234352 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:48.234417 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:48.238719 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.239744 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.239755 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.239759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.239931 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.240368 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.240373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.240376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.240917 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:48.241243 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:48.241593 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 02:49:48.241822 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:49:48.242990 16222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:48.360208 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 16099
I20260929 02:49:48.380270 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 16233
Sep 29 02:49:48 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:49:48.396200 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:49:48.381022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52020 (local address 127.0.97.3:42419)
0929 02:49:48.381247 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:49:48.381253 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:49:48.381275 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:49:48.381447 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:49:48.381451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:49:48.381519 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:49:48.381631 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:49:48.382984 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.383747 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.385229 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:49:48.385634 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:49:48.385772 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:49:48.390918 (+ 5146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:49:48.390931 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:49:48.390948 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:49:48.390984 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:49:48.393762 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.394461 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.394468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.394470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.394595 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:49:48.394926 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:49:48.394929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:49:48.394931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:49:48.395295 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:49:48.395512 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:49:48.395768 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 02:49:48.395940 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260929 02:49:48.397023 16356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:48.489151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 16233
I20260929 02:49:48.505924 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 15872
I20260929 02:49:48.507076 15934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:49:48.619542 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 15872
I20260929 02:50:08.642455 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41755
--webserver_interface=127.0.97.62
--webserver_port=37937
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:08.996073 16411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:08.996387 16411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:08.996477 16411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:09.003996 16411 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:50:09.004117 16411 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:50:09.004156 16411 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:50:09.004200 16411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:09.004248 16411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:09.004318 16411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:09.004365 16411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:09.004412 16411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:09.014953 16411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:09.016258 16411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:09.017719 16411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:09.043152 16418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:09.043192 16417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:09.044018 16420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:09.044019 16411 server_base.cc:1034] running on GCE node
I20260929 02:50:09.045228 16411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:09.047098 16411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:09.048399 16411 hybrid_clock.cc:648] HybridClock initialized: now 1790650209048328 us; error 37 us; skew 500 ppm
I20260929 02:50:09.048906 16411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:09 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:09.052914 16411 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:50:09.054775 16411 webserver.cc:460] Webserver started at http://127.0.97.62:37937/ using document root <none> and password file <none>
I20260929 02:50:09.055621 16411 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:09.055775 16411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:09.062211 16411 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:50:09.065755 16427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:09.067644 16411 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:50:09.067894 16411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "18fa4366f41f4bf399568697d521ebbf"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-6tjh"
server_key: "999d6bb2a2e2c404b016ecef8a2c3ffa"
server_key_iv: "f5f91fe75bca4fb803ceabe9485abae8"
server_key_version: "encryptionkey@0"
I20260929 02:50:09.069314 16411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:09.122496 16411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:09.128060 16411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:09.128415 16411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:09.150607 16411 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41755
I20260929 02:50:09.150651 16478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41755 every 8 connection(s)
I20260929 02:50:09.152321 16411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:09.158571 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 16411
I20260929 02:50:09.159626 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46133
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41755
--webserver_port=34249
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:50:09.160753 16479 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:50:09.166501 16479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Bootstrap starting.
I20260929 02:50:09.188287 16479 log.cc:824] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:09.201146 16479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:50:09.201777 16479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Bootstrap complete.
I20260929 02:50:09.211532 16479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:09.212423 16479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fa4366f41f4bf399568697d521ebbf, State: Initialized, Role: FOLLOWER
I20260929 02:50:09.213119 16479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:09.213393 16479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:09.213589 16479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:09.213788 16479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:50:09.217965 16479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:09.218546 16479 leader_election.cc:302] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18fa4366f41f4bf399568697d521ebbf; no voters:
I20260929 02:50:09.219226 16479 leader_election.cc:288] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 02:50:09.219372 16483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:50:09.220715 16483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 LEADER]: Becoming Leader. State: Replica: 18fa4366f41f4bf399568697d521ebbf, State: Running, Role: LEADER
I20260929 02:50:09.221345 16483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:09.221635 16479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:09.223994 16485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fa4366f41f4bf399568697d521ebbf. Latest consensus state: current_term: 2 leader_uuid: "18fa4366f41f4bf399568697d521ebbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } } }
I20260929 02:50:09.224452 16485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:09.225076 16484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "18fa4366f41f4bf399568697d521ebbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } } }
I20260929 02:50:09.225435 16484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:09.234133 16490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:09.245474 16490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:09.247617 16490 catalog_manager.cc:1259] Loaded cluster ID: 99e89ef34e9a4c3abc4593b657e39db6
I20260929 02:50:09.247884 16490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:09.253568 16490 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:09.256049 16490 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Loaded TSK: 0
I20260929 02:50:09.301541 16490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Generated new TSK 1
I20260929 02:50:09.302109 16490 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 02:50:09.564918 16481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:09.565233 16481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:09.565402 16481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:09.573202 16481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:09.573333 16481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:09.573513 16481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:50:09.585104 16481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34249
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:09.586519 16481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:09.587924 16481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:09.601845 16506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:09.601857 16507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:09.602453 16509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:09.603180 16481 server_base.cc:1034] running on GCE node
I20260929 02:50:09.604023 16481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:09.605456 16481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:09.606729 16481 hybrid_clock.cc:648] HybridClock initialized: now 1790650209606658 us; error 71 us; skew 500 ppm
I20260929 02:50:09.607157 16481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:09 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:09.618815 16481 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:50:09.621014 16481 webserver.cc:460] Webserver started at http://127.0.97.1:34249/ using document root <none> and password file <none>
I20260929 02:50:09.622191 16481 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:09.622386 16481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:09.629719 16481 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 02:50:09.633517 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:09.635454 16481 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:50:09.635661 16481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "efb83a766ad747d284490367505cd262"
format_stamp: "Formatted at 2026-09-29 02:49:46 on dist-test-slave-6tjh"
server_key: "7ea1af559df370a9f99f21bd5c7acd93"
server_key_iv: "456146da57ee42fd90257c2e42e7fb5f"
server_key_version: "encryptionkey@0"
I20260929 02:50:09.637122 16481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:09.662322 16481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:09.668269 16481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:09.668658 16481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:09.669966 16481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:09.671692 16481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:09.671804 16481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:09.671914 16481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:09.671988 16481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:09.708685 16481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46133
I20260929 02:50:09.708732 16628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46133 every 8 connection(s)
I20260929 02:50:09.710357 16481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:09.720067 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 16481
I20260929 02:50:09.720806 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33645
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41755
--webserver_port=36559
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:50:09 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:09.742141 16632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.713809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:52837)
0929 02:50:09.714680 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:09.714727 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:09.716623 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:09.718125 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:09.718133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:09.718687 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:09.719263 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.719281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.721538 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.721547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.723909 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:09.723930 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.727079 (+ 3149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.727090 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:09.727482 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.728354 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:09.728412 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:09.731663 (+ 3251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:09.737086 (+ 5423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:09.737094 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:09.737108 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:09.737630 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:09.738280 (+ 650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:09.738284 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:09.738286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:09.738515 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:09.739632 (+ 1117us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:09.739638 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:09.740009 (+ 371us) client_negotiation.cc:769] Sending connection context
0929 02:50:09.740256 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 02:50:09.740765 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":361,"threads_started":1}
I20260929 02:50:09.742559 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:09.713493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52837 (local address 127.0.97.62:41755)
0929 02:50:09.714077 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:09.714090 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:09.714876 (+ 786us) server_negotiation.cc:407] Connection header received
0929 02:50:09.717201 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:09.717214 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:09.717620 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:09.717943 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:09.719557 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.721300 (+ 1743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.724163 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:09.726710 (+ 2547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:09.727157 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:09.732032 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:09.732048 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:09.732075 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:09.732136 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:09.736868 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:09.737808 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:09.737821 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:09.737832 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:09.738093 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:09.738659 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:09.738667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:09.738673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:09.739457 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:09.739668 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:09.740578 (+ 910us) server_negotiation.cc:299] Negotiation successful
0929 02:50:09.741106 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
I20260929 02:50:09.746023 16629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41755
I20260929 02:50:09.746315 16629 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:09.747056 16629 heartbeater.cc:511] Master 127.0.97.62:41755 requested a full tablet report, sending...
I20260929 02:50:09.749475 16444 ts_manager.cc:194] Registered new tserver with Master: efb83a766ad747d284490367505cd262 (127.0.97.1:46133)
I20260929 02:50:09.750976 16444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52837
W20260929 02:50:10.069226 16633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:10.069578 16633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:10.069756 16633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:10.077368 16633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:10.077531 16633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:10.077714 16633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:50:10.088451 16633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36559
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:10.090003 16633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:10.091444 16633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:10.105798 16638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:10.105700 16641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:10.105731 16639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:10.106395 16633 server_base.cc:1034] running on GCE node
I20260929 02:50:10.107197 16633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:10.108319 16633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:10.109520 16633 hybrid_clock.cc:648] HybridClock initialized: now 1790650210109471 us; error 53 us; skew 500 ppm
I20260929 02:50:10.109845 16633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:10 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:10.113236 16633 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:50:10.114936 16633 webserver.cc:460] Webserver started at http://127.0.97.2:36559/ using document root <none> and password file <none>
I20260929 02:50:10.115733 16633 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:10.115892 16633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:10.122547 16633 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 02:50:10.127363 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:10.129364 16633 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 02:50:10.129560 16633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "88a23a0e60446031772dd0a9b9dbc963"
server_key_iv: "893da1cfc2cb5f4b74d2e36e1b3ea384"
server_key_version: "encryptionkey@0"
I20260929 02:50:10.130688 16633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:10.147923 16633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:10.153177 16633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:10.153468 16633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:10.154713 16633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:10.156361 16633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:10.156463 16633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:10.156589 16633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:10.156665 16633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:10.193923 16633 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33645
I20260929 02:50:10.193996 16760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33645 every 8 connection(s)
I20260929 02:50:10.195374 16633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:10.200960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 16633
I20260929 02:50:10.201797 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42419
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41755
--webserver_port=38473
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 02:50:10 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:10.222256 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.198479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55993 (local address 127.0.97.62:41755)
0929 02:50:10.198667 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.198672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.199698 (+ 1026us) server_negotiation.cc:407] Connection header received
0929 02:50:10.202559 (+ 2861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.202564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.202663 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.202768 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.205214 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.206017 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.209057 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.209512 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.209692 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.214360 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:10.214370 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:10.214378 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:10.214430 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:10.218316 (+ 3886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.219194 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.219202 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.219209 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.219369 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.219906 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.219920 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.219926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.220365 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:10.220527 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:10.221457 (+ 930us) server_negotiation.cc:299] Negotiation successful
0929 02:50:10.221785 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:50:10.223351 16763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.198782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:55993)
0929 02:50:10.199516 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.199556 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.202192 (+ 2636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.203107 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.203124 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.204026 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:10.204965 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.204997 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.206210 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.206219 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.208829 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.208853 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.209832 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.209843 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.210477 (+ 634us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.211258 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:10.211283 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:10.214039 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:10.218512 (+ 4473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:10.218521 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:10.218536 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:10.219041 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:10.219519 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:10.219523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:10.219526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:10.219757 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:10.220518 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:10.220526 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:10.221023 (+ 497us) client_negotiation.cc:769] Sending connection context
0929 02:50:10.221301 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 02:50:10.221905 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":288,"threads_started":1}
I20260929 02:50:10.225369 16761 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41755
I20260929 02:50:10.225739 16761 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:10.226552 16761 heartbeater.cc:511] Master 127.0.97.62:41755 requested a full tablet report, sending...
I20260929 02:50:10.228279 16444 ts_manager.cc:194] Registered new tserver with Master: fc2e3d84f7d549b1851785d6c4dfbb0a (127.0.97.2:33645)
I20260929 02:50:10.229682 16444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55993
W20260929 02:50:10.556205 16764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:10.556535 16764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:10.556742 16764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:10.564293 16764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:10.564436 16764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:10.564622 16764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:50:10.574944 16764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38473
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:10.576251 16764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:10.577708 16764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:10.591487 16769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:10.591406 16772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:10.591379 16770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:10.592180 16764 server_base.cc:1034] running on GCE node
I20260929 02:50:10.592927 16764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:10.594012 16764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:10.595252 16764 hybrid_clock.cc:648] HybridClock initialized: now 1790650210595204 us; error 49 us; skew 500 ppm
I20260929 02:50:10.595615 16764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:10 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:10.599133 16764 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:50:10.600991 16764 webserver.cc:460] Webserver started at http://127.0.97.3:38473/ using document root <none> and password file <none>
I20260929 02:50:10.601809 16764 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:10.602007 16764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:10.608361 16764 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 02:50:10.611771 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:10.613348 16764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:50:10.613551 16764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "4ebdcb76cb574884962e675c91800de6"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "3fcb7d65ce6aec6a33ab90c49a623702"
server_key_iv: "2224863ef9e50b17265996344a7f0acf"
server_key_version: "encryptionkey@0"
I20260929 02:50:10.614591 16764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:10.635106 16764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:10.639837 16764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:10.640138 16764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:10.641435 16764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:10.643081 16764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:10.643195 16764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:10.643332 16764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:10.643473 16764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:10.678748 16764 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42419
I20260929 02:50:10.678794 16891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42419 every 8 connection(s)
I20260929 02:50:10.680331 16764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:10.682348 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 16764
Sep 29 02:50:10 dist-test-slave-6tjh krb5kdc[15859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:10.699120 16631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.683280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.0.97.62:41755)
0929 02:50:10.683478 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.683486 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.684088 (+ 602us) server_negotiation.cc:407] Connection header received
0929 02:50:10.684262 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.684269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.684340 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.684460 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.686133 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.686902 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.688834 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.689609 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.689751 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.693189 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:10.693196 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:10.693201 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:10.693253 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:10.696064 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.696870 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.696875 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.696880 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.697028 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.697497 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.697501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.697504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.697865 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:10.698098 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:10.698484 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 02:50:10.698744 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:50:10.704284 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.683308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57121 (local address 127.0.97.62:41755)
0929 02:50:10.684246 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.684251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.685028 (+ 777us) server_negotiation.cc:407] Connection header received
0929 02:50:10.686860 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.686868 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.686945 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.687050 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.689137 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.690324 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.692956 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.693683 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.693834 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.697779 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:10.697784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:10.697787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:10.697820 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:10.701303 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.702141 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.702146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.702148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.702263 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.702649 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.702652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.702654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.702936 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:10.703095 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:10.703802 (+ 707us) server_negotiation.cc:299] Negotiation successful
0929 02:50:10.703990 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":194,"server-negotiator.queue_time_us":798,"thread_start_us":459,"threads_started":1}
I20260929 02:50:10.705367 16896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.683930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:57121)
0929 02:50:10.684867 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.684911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.686608 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.687372 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.687384 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.688154 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:10.688963 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.688983 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.690514 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.690521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.692772 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.692791 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.693958 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.693967 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.694315 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.694961 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:10.695001 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:10.697522 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:10.701455 (+ 3933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:10.701462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:10.701477 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:10.702018 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:10.702354 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:10.702357 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:10.702359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:10.702554 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:10.703099 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:10.703105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:10.703478 (+ 373us) client_negotiation.cc:769] Sending connection context
0929 02:50:10.703721 (+ 243us) client_negotiation.cc:240] Negotiation successful
0929 02:50:10.704132 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":393,"threads_started":1}
I20260929 02:50:10.706925 16892 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41755
I20260929 02:50:10.707245 16892 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:10.708062 16892 heartbeater.cc:511] Master 127.0.97.62:41755 requested a full tablet report, sending...
I20260929 02:50:10.709816 16444 ts_manager.cc:194] Registered new tserver with Master: 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419)
I20260929 02:50:10.710690 16444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57121
I20260929 02:50:10.714215 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:50:10.721068 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.715111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.97.62:41755)
0929 02:50:10.715276 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.715281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.715808 (+ 527us) server_negotiation.cc:407] Connection header received
0929 02:50:10.715980 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.715984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.716117 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.716251 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:10.716815 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.717560 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.719142 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.719679 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.719780 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.720267 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:10.720496 (+ 229us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 02:50:10.720868 (+ 372us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53188: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 02:50:10.721213 16895 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53188: token expired
W20260929 02:50:10.721748 16368 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:41755)
I20260929 02:50:10.735491 16895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.722262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.0.97.62:41755)
0929 02:50:10.722499 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.722505 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.722523 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:50:10.722708 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.722711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.722773 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.722866 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:10.724308 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.725110 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.726417 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.726813 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.726938 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.729648 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:10.729653 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:10.729656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:10.729690 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:10.732511 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.733314 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.733321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.733324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.733483 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:10.733881 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:10.733886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:10.733889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:10.734231 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:10.734536 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:10.734854 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 02:50:10.735102 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260929 02:50:10.739356 16444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:50:10.747006 16444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:50:10.754516 16629 heartbeater.cc:503] Master 127.0.97.62:41755 was elected leader, sending a full tablet report...
I20260929 02:50:10.771746 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.761482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33645 (local address 127.0.0.1:51820)
0929 02:50:10.762329 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.762369 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.762517 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.763547 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.763555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.763589 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:10.763992 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.764007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.766339 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.766347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.769050 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.769068 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.770625 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.770632 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.770897 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.770939 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:50:10.771074 (+ 135us) client_negotiation.cc:240] Negotiation successful
0929 02:50:10.771291 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":45,"thread_start_us":420,"threads_started":1}
I20260929 02:50:10.772758 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.761531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.97.2:33645)
0929 02:50:10.762265 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.762274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.762467 (+ 193us) server_negotiation.cc:407] Connection header received
0929 02:50:10.762716 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.762724 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.763055 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.763263 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:10.764240 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.766162 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.769231 (+ 3069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.770392 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.770547 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.770764 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:10.772177 (+ 1413us) server_negotiation.cc:299] Negotiation successful
0929 02:50:10.772371 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":469,"threads_started":1}
I20260929 02:50:10.773636 16904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.762762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46133 (local address 127.0.0.1:39550)
0929 02:50:10.763374 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.763398 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.765001 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.766327 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.766332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.766362 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:10.766848 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.766859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.768887 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.768893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.770945 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.770960 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.772818 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.772828 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.772997 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.773033 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:50:10.773105 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 02:50:10.773238 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":29,"thread_start_us":282,"threads_started":1}
I20260929 02:50:10.774175 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.763116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39550 (local address 127.0.97.1:46133)
0929 02:50:10.765647 (+ 2531us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.765656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.765689 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:50:10.765794 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.765802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.766067 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.766900 (+ 833us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:10.767127 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.768742 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.771111 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.772580 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.773353 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.773592 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:10.773671 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 02:50:10.773852 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2342,"thread_start_us":2211,"threads_started":1}
I20260929 02:50:10.774618 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.763371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42419 (local address 127.0.0.1:43560)
0929 02:50:10.766516 (+ 3145us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:10.766541 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:10.766687 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:10.767262 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:10.767266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:10.767292 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:10.767929 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.767941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.769770 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.769779 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.771651 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:10.771664 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.773850 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.773856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:10.774039 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.774074 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:50:10.774173 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:50:10.774302 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2971,"thread_start_us":2821,"threads_started":1}
I20260929 02:50:10.776141 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:10.763940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.0.97.3:42419)
0929 02:50:10.766023 (+ 2083us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:10.766032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:10.766663 (+ 631us) server_negotiation.cc:407] Connection header received
0929 02:50:10.766830 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:10.766840 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:10.767062 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:10.767252 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:10.768156 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.769622 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.771876 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:10.773349 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:10.773477 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:10.773706 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:10.774232 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 02:50:10.775808 (+ 1576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1920,"thread_start_us":1762,"threads_started":1}
I20260929 02:50:10.777637 16696 tablet_service.cc:1467] Processing CreateTablet for tablet a0506dd545f54c0f8d49136e3bf9b6a4 (DEFAULT_TABLE table=test-table [id=630e670720414083ace19679fbca8316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:10.778214 16564 tablet_service.cc:1467] Processing CreateTablet for tablet a0506dd545f54c0f8d49136e3bf9b6a4 (DEFAULT_TABLE table=test-table [id=630e670720414083ace19679fbca8316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:10.779170 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0506dd545f54c0f8d49136e3bf9b6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:10.779570 16564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0506dd545f54c0f8d49136e3bf9b6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:10.779609 16827 tablet_service.cc:1467] Processing CreateTablet for tablet a0506dd545f54c0f8d49136e3bf9b6a4 (DEFAULT_TABLE table=test-table [id=630e670720414083ace19679fbca8316]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:10.780995 16827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0506dd545f54c0f8d49136e3bf9b6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:10.794301 16908 tablet_bootstrap.cc:492] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Bootstrap starting.
I20260929 02:50:10.797712 16909 tablet_bootstrap.cc:492] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Bootstrap starting.
I20260929 02:50:10.798664 16910 tablet_bootstrap.cc:492] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Bootstrap starting.
I20260929 02:50:10.798797 16908 tablet_bootstrap.cc:654] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:10.800396 16908 log.cc:824] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:10.802287 16909 tablet_bootstrap.cc:654] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:10.802846 16910 tablet_bootstrap.cc:654] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:10.803416 16909 log.cc:824] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:10.804490 16910 log.cc:824] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:10.806761 16908 tablet_bootstrap.cc:492] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: No bootstrap required, opened a new log
I20260929 02:50:10.806923 16909 tablet_bootstrap.cc:492] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: No bootstrap required, opened a new log
I20260929 02:50:10.807108 16908 ts_tablet_manager.cc:1397] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260929 02:50:10.807235 16909 ts_tablet_manager.cc:1397] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 02:50:10.807893 16910 tablet_bootstrap.cc:492] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: No bootstrap required, opened a new log
I20260929 02:50:10.808271 16910 ts_tablet_manager.cc:1397] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 02:50:10.815181 16908 raft_consensus.cc:348] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:10.815645 16908 raft_consensus.cc:374] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:10.815745 16908 raft_consensus.cc:729] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb83a766ad747d284490367505cd262, State: Initialized, Role: FOLLOWER
I20260929 02:50:10.816409 16908 consensus_queue.cc:260] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:10.818305 16908 ts_tablet_manager.cc:1428] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 02:50:10.819765 16909 raft_consensus.cc:348] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:10.820190 16909 raft_consensus.cc:374] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:10.820317 16909 raft_consensus.cc:729] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc2e3d84f7d549b1851785d6c4dfbb0a, State: Initialized, Role: FOLLOWER
I20260929 02:50:10.820806 16910 raft_consensus.cc:348] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:10.821028 16909 consensus_queue.cc:260] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:10.821280 16910 raft_consensus.cc:374] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:10.821390 16910 raft_consensus.cc:729] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ebdcb76cb574884962e675c91800de6, State: Initialized, Role: FOLLOWER
I20260929 02:50:10.822007 16910 consensus_queue.cc:260] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:10.824299 16892 heartbeater.cc:503] Master 127.0.97.62:41755 was elected leader, sending a full tablet report...
I20260929 02:50:10.825634 16910 ts_tablet_manager.cc:1428] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260929 02:50:10.826983 16761 heartbeater.cc:503] Master 127.0.97.62:41755 was elected leader, sending a full tablet report...
I20260929 02:50:10.827680 16909 ts_tablet_manager.cc:1428] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Time spent starting tablet: real 0.020s user 0.018s sys 0.003s
W20260929 02:50:10.932088 16893 tablet.cc:2369] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:50:10.947646 16762 tablet.cc:2369] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:50:10.963126 16630 tablet.cc:2369] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:11.187790 16914 raft_consensus.cc:482] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:50:11.188134 16914 raft_consensus.cc:504] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:11.190064 16914 leader_election.cc:288] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc2e3d84f7d549b1851785d6c4dfbb0a (127.0.97.2:33645), 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419)
I20260929 02:50:11.198860 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.191447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47899 (local address 127.0.97.3:42419)
0929 02:50:11.191721 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.191731 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.192195 (+ 464us) server_negotiation.cc:407] Connection header received
0929 02:50:11.192513 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.192518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.192638 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.192752 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:11.193412 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.194281 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.196266 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.197728 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.197884 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.198016 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:11.198400 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 02:50:11.198567 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260929 02:50:11.199007 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.191311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42419 (local address 127.0.97.1:47899)
0929 02:50:11.192036 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:11.192068 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:11.192217 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:11.192764 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:11.192767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:11.192791 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:11.193216 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.193229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.194426 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.194435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.196085 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.196099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.197979 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.197986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.198185 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.198227 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:50:11.198358 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 02:50:11.198538 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":25,"thread_start_us":311,"threads_started":1}
I20260929 02:50:11.199162 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.190547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48263 (local address 127.0.97.2:33645)
0929 02:50:11.190787 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.190797 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.191246 (+ 449us) server_negotiation.cc:407] Connection header received
0929 02:50:11.191536 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.191547 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.191672 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.191805 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:11.192512 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.193609 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.196481 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.197957 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.198119 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.198230 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:11.198682 (+ 452us) server_negotiation.cc:299] Negotiation successful
0929 02:50:11.198818 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:50:11.200337 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.190448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33645 (local address 127.0.97.1:48263)
0929 02:50:11.191119 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:11.191137 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:11.191333 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:11.191843 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:11.191849 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:11.191888 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:11.192320 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.192332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.193790 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.193796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.196290 (+ 2494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:11.196306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.198219 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.198226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:11.198451 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.198489 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 02:50:11.199442 (+ 953us) client_negotiation.cc:240] Negotiation successful
0929 02:50:11.199620 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":27,"thread_start_us":328,"threads_started":1}
I20260929 02:50:11.200445 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0506dd545f54c0f8d49136e3bf9b6a4" candidate_uuid: "efb83a766ad747d284490367505cd262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebdcb76cb574884962e675c91800de6" is_pre_election: true
I20260929 02:50:11.201083 16847 raft_consensus.cc:2454] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb83a766ad747d284490367505cd262 in term 0.
I20260929 02:50:11.201484 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0506dd545f54c0f8d49136e3bf9b6a4" candidate_uuid: "efb83a766ad747d284490367505cd262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" is_pre_election: true
I20260929 02:50:11.201846 16517 leader_election.cc:302] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ebdcb76cb574884962e675c91800de6, efb83a766ad747d284490367505cd262; no voters:
I20260929 02:50:11.202206 16716 raft_consensus.cc:2454] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb83a766ad747d284490367505cd262 in term 0.
I20260929 02:50:11.202451 16914 raft_consensus.cc:2781] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:50:11.202592 16914 raft_consensus.cc:482] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:50:11.202673 16914 raft_consensus.cc:3037] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:11.206291 16914 raft_consensus.cc:504] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:11.207065 16914 leader_election.cc:288] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [CANDIDATE]: Term 1 election: Requested vote from peers fc2e3d84f7d549b1851785d6c4dfbb0a (127.0.97.2:33645), 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419)
I20260929 02:50:11.207711 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0506dd545f54c0f8d49136e3bf9b6a4" candidate_uuid: "efb83a766ad747d284490367505cd262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a"
I20260929 02:50:11.207844 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0506dd545f54c0f8d49136e3bf9b6a4" candidate_uuid: "efb83a766ad747d284490367505cd262" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ebdcb76cb574884962e675c91800de6"
I20260929 02:50:11.207954 16716 raft_consensus.cc:3037] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:11.208096 16847 raft_consensus.cc:3037] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:11.210933 16716 raft_consensus.cc:2454] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb83a766ad747d284490367505cd262 in term 1.
I20260929 02:50:11.211499 16517 leader_election.cc:302] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: efb83a766ad747d284490367505cd262, fc2e3d84f7d549b1851785d6c4dfbb0a; no voters:
I20260929 02:50:11.211642 16847 raft_consensus.cc:2454] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb83a766ad747d284490367505cd262 in term 1.
I20260929 02:50:11.211937 16914 raft_consensus.cc:2781] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:11.212617 16914 raft_consensus.cc:686] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 1 LEADER]: Becoming Leader. State: Replica: efb83a766ad747d284490367505cd262, State: Running, Role: LEADER
I20260929 02:50:11.213251 16914 consensus_queue.cc:237] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } }
I20260929 02:50:11.218554 16443 catalog_manager.cc:5495] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 reported cstate change: term changed from 0 to 1, leader changed from <none> to efb83a766ad747d284490367505cd262 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "efb83a766ad747d284490367505cd262" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb83a766ad747d284490367505cd262" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 } health_report { overall_health: UNKNOWN } } }
I20260929 02:50:11.342460 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.336379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39562 (local address 127.0.97.1:46133)
0929 02:50:11.336963 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.336970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.336991 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:50:11.337235 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.337239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.337317 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.337485 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:11.337885 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.338574 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.340085 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.340490 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.340695 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.341220 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:11.341808 (+ 588us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:11.341976 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:11.342028 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 02:50:11.342113 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":311,"threads_started":1}
I20260929 02:50:11.351081 16716 raft_consensus.cc:1261] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 FOLLOWER]: Refusing update from remote peer efb83a766ad747d284490367505cd262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:11.351102 16847 raft_consensus.cc:1261] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Refusing update from remote peer efb83a766ad747d284490367505cd262: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:11.352164 16919 consensus_queue.cc:1035] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ebdcb76cb574884962e675c91800de6" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:11.352699 16914 consensus_queue.cc:1035] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:11.384645 16443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:11.384913 16443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:11.388336 16443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Sending DeleteTablet for 3 replicas of tablet a0506dd545f54c0f8d49136e3bf9b6a4
I20260929 02:50:11.389799 16696 tablet_service.cc:1514] Processing DeleteTablet for tablet a0506dd545f54c0f8d49136e3bf9b6a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51820
I20260929 02:50:11.390000 16564 tablet_service.cc:1514] Processing DeleteTablet for tablet a0506dd545f54c0f8d49136e3bf9b6a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39550
I20260929 02:50:11.390008 16827 tablet_service.cc:1514] Processing DeleteTablet for tablet a0506dd545f54c0f8d49136e3bf9b6a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:11 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43560
I20260929 02:50:11.390928 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 16481
I20260929 02:50:11.401381 16938 tablet_replica.cc:330] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: stopping tablet replica
I20260929 02:50:11.401968 16938 raft_consensus.cc:2229] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:50:11.402804 16938 raft_consensus.cc:2258] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:11.404803 16936 tablet_replica.cc:330] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: stopping tablet replica
I20260929 02:50:11.405686 16936 raft_consensus.cc:2229] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:11.406252 16936 raft_consensus.cc:2258] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:11.408196 16938 ts_tablet_manager.cc:1905] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:11.408470 16937 tablet_replica.cc:330] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: stopping tablet replica
I20260929 02:50:11.408983 16937 raft_consensus.cc:2229] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:11.409416 16937 raft_consensus.cc:2258] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:11.412348 16937 ts_tablet_manager.cc:1905] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:11.417743 16938 ts_tablet_manager.cc:1918] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:11.417973 16938 log.cc:1196] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/a0506dd545f54c0f8d49136e3bf9b6a4
I20260929 02:50:11.418655 16938 ts_tablet_manager.cc:1939] T a0506dd545f54c0f8d49136e3bf9b6a4 P efb83a766ad747d284490367505cd262: Deleting consensus metadata
I20260929 02:50:11.420130 16937 ts_tablet_manager.cc:1918] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:11.420303 16937 log.cc:1196] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/a0506dd545f54c0f8d49136e3bf9b6a4
I20260929 02:50:11.420905 16937 ts_tablet_manager.cc:1939] T a0506dd545f54c0f8d49136e3bf9b6a4 P 4ebdcb76cb574884962e675c91800de6: Deleting consensus metadata
I20260929 02:50:11.420884 16428 catalog_manager.cc:4841] TS efb83a766ad747d284490367505cd262 (127.0.97.1:46133): tablet a0506dd545f54c0f8d49136e3bf9b6a4 (table test-table [id=630e670720414083ace19679fbca8316]) successfully deleted
I20260929 02:50:11.421851 16936 ts_tablet_manager.cc:1905] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:11.422791 16428 catalog_manager.cc:4841] TS 4ebdcb76cb574884962e675c91800de6 (127.0.97.3:42419): tablet a0506dd545f54c0f8d49136e3bf9b6a4 (table test-table [id=630e670720414083ace19679fbca8316]) successfully deleted
I20260929 02:50:11.425343 16923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.392000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39568 (local address 127.0.97.1:46133)
0929 02:50:11.392222 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.392233 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.404008 (+ 11775us) server_negotiation.cc:407] Connection header received
0929 02:50:11.404234 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.404242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.404347 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.404467 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:11.407952 (+ 3485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.409049 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.411137 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.411843 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.412012 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.416321 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:11.416336 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:11.416355 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:11.416402 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:11.421677 (+ 5275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:11.422882 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:11.422893 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:11.422899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:11.423210 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:11.423639 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:11.423644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:11.423647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:11.424024 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:11.424374 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:11.424700 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 02:50:11.424975 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:50:11.426244 16624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:11.430905 16936 ts_tablet_manager.cc:1918] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:11.431144 16936 log.cc:1196] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/a0506dd545f54c0f8d49136e3bf9b6a4
I20260929 02:50:11.431916 16936 ts_tablet_manager.cc:1939] T a0506dd545f54c0f8d49136e3bf9b6a4 P fc2e3d84f7d549b1851785d6c4dfbb0a: Deleting consensus metadata
I20260929 02:50:11.434145 16428 catalog_manager.cc:4841] TS fc2e3d84f7d549b1851785d6c4dfbb0a (127.0.97.2:33645): tablet a0506dd545f54c0f8d49136e3bf9b6a4 (table test-table [id=630e670720414083ace19679fbca8316]) successfully deleted
I20260929 02:50:11.527405 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 16481
I20260929 02:50:11.546377 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 16633
I20260929 02:50:11.562729 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.547355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.0.97.2:33645)
0929 02:50:11.547571 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.547599 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.547620 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:50:11.547686 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.547690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.547753 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.547869 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:11.549309 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.550066 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.551429 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.551841 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.551974 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.554645 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:11.554660 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:11.554677 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:11.554713 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:11.558707 (+ 3994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:11.559959 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:11.559973 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:11.559987 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:11.560187 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:11.560701 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:11.560706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:11.560710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:11.561302 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:11.561638 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:11.562018 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 02:50:11.562305 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 02:50:11.563606 16756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:11.656980 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 16633
I20260929 02:50:11.674096 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 16764
I20260929 02:50:11.690887 16907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:11.674845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.0.97.3:42419)
0929 02:50:11.675025 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:11.675030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:11.675103 (+ 73us) server_negotiation.cc:407] Connection header received
0929 02:50:11.675325 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:11.675330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:11.675489 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:11.675592 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:11.677234 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.678034 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.680108 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:11.680762 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:11.680908 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:11.684497 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:11.684509 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:11.684523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:11.684666 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:11.687882 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:11.688837 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:11.688844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:11.688847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:11.689011 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:11.689428 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:11.689432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:11.689435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:11.689809 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:11.690054 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:11.690426 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 02:50:11.690595 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 02:50:11.691636 16887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:11.785452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 16764
I20260929 02:50:11.803133 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 16411
I20260929 02:50:11.804374 16474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:11.922065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 16411
I20260929 02:50:31.944783 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41755
--webserver_interface=127.0.97.62
--webserver_port=37937
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:32.289877 16944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:32.290256 16944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:32.290422 16944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:32.297911 16944 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 02:50:32.298070 16944 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 02:50:32.298167 16944 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 02:50:32.298264 16944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:32.298367 16944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:32.298491 16944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:32.298585 16944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:32.298668 16944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:32.308954 16944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=37937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:32.310331 16944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:32.311733 16944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:32.323995 16949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:32.324082 16950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:32.324026 16952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:32.325412 16944 server_base.cc:1034] running on GCE node
I20260929 02:50:32.326246 16944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:32.327739 16944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:32.328984 16944 hybrid_clock.cc:648] HybridClock initialized: now 1790650232328911 us; error 50 us; skew 500 ppm
I20260929 02:50:32.329372 16944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:32 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:32.338317 16944 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:50:32.340085 16944 webserver.cc:460] Webserver started at http://127.0.97.62:37937/ using document root <none> and password file <none>
I20260929 02:50:32.340991 16944 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:32.341126 16944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:32.346941 16944 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 02:50:32.350198 16959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:32.351902 16944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:50:32.352144 16944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "18fa4366f41f4bf399568697d521ebbf"
format_stamp: "Formatted at 2026-09-29 02:49:45 on dist-test-slave-6tjh"
server_key: "999d6bb2a2e2c404b016ecef8a2c3ffa"
server_key_iv: "f5f91fe75bca4fb803ceabe9485abae8"
server_key_version: "encryptionkey@0"
I20260929 02:50:32.353298 16944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:32.392519 16944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:32.397723 16944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:32.398053 16944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:32.417685 16944 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41755
I20260929 02:50:32.417747 17010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41755 every 8 connection(s)
I20260929 02:50:32.419268 16944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:32.420666 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 16944
I20260929 02:50:32.421501 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46133
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41755
--webserver_port=34249
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:50:32.427356 17011 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:50:32.431118 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Bootstrap starting.
I20260929 02:50:32.454244 17011 log.cc:824] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:32.471969 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:50:32.472661 17011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Bootstrap complete.
I20260929 02:50:32.481966 17011 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:32.482890 17011 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18fa4366f41f4bf399568697d521ebbf, State: Initialized, Role: FOLLOWER
I20260929 02:50:32.483559 17011 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:32.483798 17011 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:32.483929 17011 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:32.484109 17011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 2 FOLLOWER]: Advancing to term 3
I20260929 02:50:32.488092 17011 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:32.488765 17011 leader_election.cc:302] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18fa4366f41f4bf399568697d521ebbf; no voters:
I20260929 02:50:32.489459 17011 leader_election.cc:288] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 02:50:32.489639 17015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 3 FOLLOWER]: Leader election won for term 3
I20260929 02:50:32.490844 17015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [term 3 LEADER]: Becoming Leader. State: Replica: 18fa4366f41f4bf399568697d521ebbf, State: Running, Role: LEADER
I20260929 02:50:32.491421 17015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } }
I20260929 02:50:32.491768 17011 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:32.497720 17016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "18fa4366f41f4bf399568697d521ebbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } } }
I20260929 02:50:32.498242 17016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:32.498862 17017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18fa4366f41f4bf399568697d521ebbf. Latest consensus state: current_term: 3 leader_uuid: "18fa4366f41f4bf399568697d521ebbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18fa4366f41f4bf399568697d521ebbf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41755 } } }
I20260929 02:50:32.499215 17017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:32.503245 17023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:32.514205 17023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:32.516214 17023 catalog_manager.cc:1259] Loaded cluster ID: 99e89ef34e9a4c3abc4593b657e39db6
I20260929 02:50:32.516350 17023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:32.522976 17023 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:32.525468 17023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Loaded TSK: 0
I20260929 02:50:32.525676 17023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 18fa4366f41f4bf399568697d521ebbf: Loaded TSK: 1
F20260929 02:50:32.527163 17023 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb14d6119ad google::LogMessage::Fail() at ??:0
@ 0x7fb14d61488a google::LogMessage::SendToLog() at ??:0
@ 0x7fb14d6114de google::LogMessage::Flush() at ??:0
@ 0x7fb14d612a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb1513757d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb15136473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb1630401c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb16303f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb1630cb2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb1630cb0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb1631155e9 std::function<>::operator()() at ??:0
@ 0x7fb163043cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb163041c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb1630c9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb1630c9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb15025d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb15026091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb150260799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fb15023beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb1535156db start_thread at ??:0
@ 0x7fb14c4ab61f clone at ??:0
W20260929 02:50:32.851842 17013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:32.852159 17013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:32.852255 17013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:32.859884 17013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:32.860006 17013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:32.860129 17013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:50:32.870337 17013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34249
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:32.871615 17013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:32.872961 17013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:32.886513 17062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:32.886574 17060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:32.886590 17059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:32.887422 17013 server_base.cc:1034] running on GCE node
I20260929 02:50:32.888125 17013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:32.889225 17013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:32.890488 17013 hybrid_clock.cc:648] HybridClock initialized: now 1790650232890425 us; error 54 us; skew 500 ppm
I20260929 02:50:32.890877 17013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:32 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:32.894532 17013 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:50:32.896111 17013 webserver.cc:460] Webserver started at http://127.0.97.1:34249/ using document root <none> and password file <none>
I20260929 02:50:32.896919 17013 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:32.897040 17013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:32.902837 17013 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 02:50:32.905834 17069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:32.907353 17013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:50:32.907536 17013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "efb83a766ad747d284490367505cd262"
format_stamp: "Formatted at 2026-09-29 02:49:46 on dist-test-slave-6tjh"
server_key: "7ea1af559df370a9f99f21bd5c7acd93"
server_key_iv: "456146da57ee42fd90257c2e42e7fb5f"
server_key_version: "encryptionkey@0"
I20260929 02:50:32.908802 17013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:32.932475 17013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:32.937575 17013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:32.937906 17013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:32.939010 17013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:32.940444 17013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:32.940610 17013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:32.940779 17013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:32.940896 17013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:32.977837 17013 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46133
I20260929 02:50:32.977898 17181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46133 every 8 connection(s)
I20260929 02:50:32.979539 17013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:32.984828 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.982735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:52621)
0929 02:50:32.983550 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:32.983778 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":319,"threads_started":1}
W20260929 02:50:32.985502 17182 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41755: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
I20260929 02:50:32.986392 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.986028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:53527)
0929 02:50:32.986198 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:32.986275 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:50:32.987571 17184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:32.987060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:36797)
0929 02:50:32.987294 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:32.987395 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 02:50:32.987785 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 17013
W20260929 02:50:32.988291 17182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:50:32.988644 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:33645
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41755
--webserver_port=36559
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:33.331854 17185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:33.332170 17185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:33.332269 17185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:33.339769 17185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:33.339898 17185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:33.340022 17185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:50:33.350924 17185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:33645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=36559
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:33.352232 17185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:33.353621 17185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:33.368372 17190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:33.368295 17191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:33.368513 17193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:33.369800 17185 server_base.cc:1034] running on GCE node
I20260929 02:50:33.370550 17185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:33.371686 17185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:33.372997 17185 hybrid_clock.cc:648] HybridClock initialized: now 1790650233372944 us; error 60 us; skew 500 ppm
I20260929 02:50:33.373391 17185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:33 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:33.377134 17185 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:50:33.378871 17185 webserver.cc:460] Webserver started at http://127.0.97.2:36559/ using document root <none> and password file <none>
I20260929 02:50:33.379642 17185 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:33.379796 17185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:33.385878 17185 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:50:33.389070 17200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:33.390739 17185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 02:50:33.390962 17185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "fc2e3d84f7d549b1851785d6c4dfbb0a"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "88a23a0e60446031772dd0a9b9dbc963"
server_key_iv: "893da1cfc2cb5f4b74d2e36e1b3ea384"
server_key_version: "encryptionkey@0"
I20260929 02:50:33.392032 17185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:33.417197 17185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:33.422158 17185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:33.422497 17185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:33.423710 17185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:33.425295 17185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:33.425455 17185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:33.425608 17185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:33.425705 17185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:33.461901 17185 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33645
I20260929 02:50:33.462159 17312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33645 every 8 connection(s)
I20260929 02:50:33.463686 17185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:33.464218 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 17185
I20260929 02:50:33.465123 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42419
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41755
--webserver_port=38473
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:50:33.469899 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:33.467100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:32911)
0929 02:50:33.468119 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:33.468430 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":377,"threads_started":1}
W20260929 02:50:33.470674 17313 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41755: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
I20260929 02:50:33.471840 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:33.471292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:51449)
0929 02:50:33.471547 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:33.471657 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:50:33.473078 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:33.472527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:39513)
0929 02:50:33.472763 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:33.472892 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260929 02:50:33.473354 17313 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 02:50:33.816625 17316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:33.817024 17316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:33.817255 17316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:33.824836 17316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:33.824958 17316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:33.825155 17316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:50:33.835311 17316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=38473
--tserver_master_addrs=127.0.97.62:41755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:33.836761 17316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:33.838201 17316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:33.852069 17321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:33.852205 17324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:33.852260 17322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:33.853072 17316 server_base.cc:1034] running on GCE node
I20260929 02:50:33.853701 17316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:33.854766 17316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:33.855969 17316 hybrid_clock.cc:648] HybridClock initialized: now 1790650233855928 us; error 40 us; skew 500 ppm
I20260929 02:50:33.856329 17316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:33 dist-test-slave-6tjh krb5kdc[15859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:33.859592 17316 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:50:33.861339 17316 webserver.cc:460] Webserver started at http://127.0.97.3:38473/ using document root <none> and password file <none>
I20260929 02:50:33.862076 17316 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:33.862229 17316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:33.867883 17316 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:50:33.871042 17331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:33.872684 17316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:50:33.872886 17316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "4ebdcb76cb574884962e675c91800de6"
format_stamp: "Formatted at 2026-09-29 02:49:47 on dist-test-slave-6tjh"
server_key: "3fcb7d65ce6aec6a33ab90c49a623702"
server_key_iv: "2224863ef9e50b17265996344a7f0acf"
server_key_version: "encryptionkey@0"
I20260929 02:50:33.873929 17316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:33.896663 17316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:33.901659 17316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:33.901998 17316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:33.903163 17316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:33.904719 17316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:33.904855 17316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:33.904978 17316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:33.905090 17316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:33.942811 17316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42419
I20260929 02:50:33.942894 17443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42419 every 8 connection(s)
I20260929 02:50:33.944391 17316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:33.949851 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:33.947668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:43675)
0929 02:50:33.948530 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:33.948861 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":360,"threads_started":1}
W20260929 02:50:33.950703 17444 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41755 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41755: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
I20260929 02:50:33.951817 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:33.951348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:34307)
0929 02:50:33.951598 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:33.951688 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 02:50:33.952589 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 17316
I20260929 02:50:33.953039 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:33.952527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:57939)
0929 02:50:33.952767 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:33.952856 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260929 02:50:33.953411 17444 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 02:50:33.989694 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:33.988862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:55849)
0929 02:50:33.989351 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:33.989436 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":249,"threads_started":1}
I20260929 02:50:34.474958 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:34.474049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:46971)
0929 02:50:34.474570 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:34.474670 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":238,"threads_started":1}
I20260929 02:50:34.954875 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:34.954053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:57727)
0929 02:50:34.954542 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:34.954637 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260929 02:50:34.991811 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:34.990746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:56067)
0929 02:50:34.991417 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:34.991557 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":293,"threads_started":1}
I20260929 02:50:35.476820 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:35.476026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:48713)
0929 02:50:35.476519 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:35.476642 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":246,"threads_started":1}
I20260929 02:50:35.956733 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:35.955839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:46259)
0929 02:50:35.956367 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:35.956453 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":270,"threads_started":1}
I20260929 02:50:35.993639 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:35.992813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:34617)
0929 02:50:35.993345 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:35.993451 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":294,"threads_started":1}
I20260929 02:50:36.478856 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:36.477936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:41147)
0929 02:50:36.478523 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:36.478619 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":347,"threads_started":1}
I20260929 02:50:36.958573 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:36.957722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:54505)
0929 02:50:36.958282 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:36.958373 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":312,"threads_started":1}
I20260929 02:50:36.995563 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:36.994665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:55271)
0929 02:50:36.995245 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:36.995339 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":267,"threads_started":1}
I20260929 02:50:37.480712 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:37.479911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:52473)
0929 02:50:37.480368 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:37.480462 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":234,"threads_started":1}
I20260929 02:50:37.960459 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:37.959649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:59341)
0929 02:50:37.960188 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:37.960277 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
I20260929 02:50:37.997514 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:37.996726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:53237)
0929 02:50:37.997248 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:37.997334 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
I20260929 02:50:38.482585 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:38.481645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:50393)
0929 02:50:38.482269 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:38.482372 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":337,"threads_started":1}
I20260929 02:50:38.962249 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:38.961514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:58335)
0929 02:50:38.961970 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:38.962075 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":233,"threads_started":1}
I20260929 02:50:38.999323 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:38.998524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:42935)
0929 02:50:38.999028 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:38.999153 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":264,"threads_started":1}
I20260929 02:50:39.484637 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:39.483731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:36809)
0929 02:50:39.484267 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:39.484385 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":313,"threads_started":1}
I20260929 02:50:39.964079 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:39.963273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:35329)
0929 02:50:39.963799 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:39.963908 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":322,"threads_started":1}
I20260929 02:50:40.001148 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:40.000270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:55433)
0929 02:50:40.000816 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:40.000901 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":291,"threads_started":1}
I20260929 02:50:40.486838 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:40.485759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:54757)
0929 02:50:40.486434 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:40.486546 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":310,"threads_started":1}
I20260929 02:50:40.965953 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:40.965176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:60733)
0929 02:50:40.965663 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:40.965746 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":210,"threads_started":1}
I20260929 02:50:41.003046 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:41.002276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:37279)
0929 02:50:41.002764 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:41.002852 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":260,"threads_started":1}
I20260929 02:50:41.488842 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:41.487998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:41387)
0929 02:50:41.488477 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:41.488667 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":248,"threads_started":1}
I20260929 02:50:41.967760 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:41.966972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:56429)
0929 02:50:41.967487 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:41.967582 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":264,"threads_started":1}
I20260929 02:50:42.005046 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:42.004112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:58839)
0929 02:50:42.004721 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:42.004813 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":264,"threads_started":1}
I20260929 02:50:42.490854 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:42.490017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:49795)
0929 02:50:42.490581 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:42.490671 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":279,"threads_started":1}
I20260929 02:50:42.969826 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:42.968940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:55689)
0929 02:50:42.969439 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:42.969567 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":254,"threads_started":1}
I20260929 02:50:43.006886 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:43.006068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:47985)
0929 02:50:43.006622 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:43.006710 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":313,"threads_started":1}
I20260929 02:50:43.492678 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:43.491844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:51097)
0929 02:50:43.492342 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:43.492423 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
I20260929 02:50:43.971824 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:43.971027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:54293)
0929 02:50:43.971542 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:43.971643 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":287,"threads_started":1}
I20260929 02:50:44.008898 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:44.007966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:54269)
0929 02:50:44.008500 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:44.008653 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":292,"threads_started":1}
I20260929 02:50:44.494568 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:44.493729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:58589)
0929 02:50:44.494285 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:44.494383 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":315,"threads_started":1}
I20260929 02:50:44.973908 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:44.973059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:34863)
0929 02:50:44.973546 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:44.973666 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":208,"threads_started":1}
I20260929 02:50:45.010712 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:45.009919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:39045)
0929 02:50:45.010426 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:45.010542 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":245,"threads_started":1}
I20260929 02:50:45.496508 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:45.495695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:60327)
0929 02:50:45.496226 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:45.496334 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":270,"threads_started":1}
I20260929 02:50:45.975929 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:45.975055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:43093)
0929 02:50:45.975644 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:45.975751 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":322,"threads_started":1}
I20260929 02:50:46.012759 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:46.011755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:34531)
0929 02:50:46.012322 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:46.012511 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":310,"threads_started":1}
I20260929 02:50:46.498746 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:46.497683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:60193)
0929 02:50:46.498303 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:46.498405 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":330,"threads_started":1}
I20260929 02:50:46.977910 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:46.977048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:57783)
0929 02:50:46.977617 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:46.977709 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":242,"threads_started":1}
I20260929 02:50:47.014586 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:47.013714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:49467)
0929 02:50:47.014288 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:47.014380 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":324,"threads_started":1}
I20260929 02:50:47.500475 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:47.499736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:49653)
0929 02:50:47.500202 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:47.500290 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":230,"threads_started":1}
I20260929 02:50:47.979903 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:47.979076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:57669)
0929 02:50:47.979639 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:47.979725 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260929 02:50:48.016799 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:48.015889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.1:37177)
0929 02:50:48.016383 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:48.016525 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260929 02:50:48.502671 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:48.501849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.2:43325)
0929 02:50:48.502353 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:48.502453 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":267,"threads_started":1}
I20260929 02:50:48.957590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 17013
I20260929 02:50:48.982316 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:48.981218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41755 (local address 127.0.97.3:40297)
0929 02:50:48.982020 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:48.982116 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":506,"threads_started":1}
I20260929 02:50:48.982393 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:48.958911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58132 (local address 127.0.97.1:46133)
0929 02:50:48.959652 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:48.959675 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:48.959729 (+ 54us) server_negotiation.cc:407] Connection header received
0929 02:50:48.961355 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:48.961365 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:48.961806 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:48.962112 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:48.963605 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:48.964935 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:48.966301 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:48.966997 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:48.967176 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:48.974172 (+ 6996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:48.974182 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:48.974202 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:48.974257 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:48.977655 (+ 3398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:48.978607 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:48.978615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:48.978623 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:48.978825 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:48.979329 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:48.979334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:48.979338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:48.980176 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:48.980580 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:48.981216 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 02:50:48.981912 (+ 696us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":357,"threads_started":1}
I20260929 02:50:48.984197 17177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:49.074621 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 17013
I20260929 02:50:49.092499 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 17185
I20260929 02:50:49.111824 17496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:49.093678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.0.97.2:33645)
0929 02:50:49.094283 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:49.094297 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:49.094335 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:50:49.095961 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:49.095972 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:49.096428 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:49.096816 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:49.098205 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:49.099540 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:49.101016 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:49.101763 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:49.101921 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:49.104934 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:49.104946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:49.104972 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:49.105028 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:49.108234 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:49.109229 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:49.109236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:49.109244 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:49.109434 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:49.109819 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:49.109822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:49.109824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:49.110373 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:49.110625 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:49.111042 (+ 417us) server_negotiation.cc:299] Negotiation successful
0929 02:50:49.111509 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":289,"threads_started":1}
I20260929 02:50:49.113360 17308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:49.198743 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 17185
I20260929 02:50:49.217650 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 17316
I20260929 02:50:49.237686 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:49.218763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.0.97.3:42419)
0929 02:50:49.219384 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:49.219400 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:49.219442 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:50:49.221193 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:49.221204 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:49.221666 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:49.222002 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:49.223605 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:49.225101 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:49.226619 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:49.227350 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:49.227509 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:49.230300 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:49.230309 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:49.230328 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:49.230371 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:49.233794 (+ 3423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:49.234589 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:49.234598 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:49.234606 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:49.234825 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:49.235344 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:49.235348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:49.235350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:49.235979 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:49.236227 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:49.236811 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 02:50:49.237333 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":302,"threads_started":1}
I20260929 02:50:49.239197 17439 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:49.327442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 17316
2026-09-29T02:50:49Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65137 ms)
[----------] 30 tests from SecurityITest (299794 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:50:49 dist-test-slave-6tjh krb5kdc[17502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:50:49 dist-test-slave-6tjh krb5kdc[17502](info): set up 2 sockets
Sep 29 02:50:49 dist-test-slave-6tjh krb5kdc[17502](info): commencing operation
krb5kdc: starting...
W20260929 02:50:50.452934 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:50:50 dist-test-slave-6tjh krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:50:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:50:50.717638 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37785
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:51.063584 17515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:51.063889 17515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:51.064071 17515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:51.071653 17515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:51.071774 17515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:51.071880 17515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:51.071977 17515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:51.072052 17515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:51.082496 17515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:51.083946 17515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:51.085460 17515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:51.099781 17521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:51.099844 17520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:51.099957 17515 server_base.cc:1034] running on GCE node
W20260929 02:50:51.099869 17523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:51.101440 17515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:51.103087 17515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:51.104360 17515 hybrid_clock.cc:648] HybridClock initialized: now 1790650251104315 us; error 47 us; skew 500 ppm
I20260929 02:50:51.104817 17515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:51 dist-test-slave-6tjh krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:51.108417 17515 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:50:51.110214 17515 webserver.cc:460] Webserver started at http://127.0.97.62:37879/ using document root <none> and password file <none>
I20260929 02:50:51.110941 17515 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:51.111061 17515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:51.111323 17515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:51.114562 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "9491ed1e1d32432cabf7cd7df81bdebd"
format_stamp: "Formatted at 2026-09-29 02:50:51 on dist-test-slave-6tjh"
server_key: "c79a9f16623ebd840c7c0779e13165a4"
server_key_iv: "edc74c836840e9f42e49c77e267da376"
server_key_version: "encryptionkey@0"
I20260929 02:50:51.115356 17515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "9491ed1e1d32432cabf7cd7df81bdebd"
format_stamp: "Formatted at 2026-09-29 02:50:51 on dist-test-slave-6tjh"
server_key: "c79a9f16623ebd840c7c0779e13165a4"
server_key_iv: "edc74c836840e9f42e49c77e267da376"
server_key_version: "encryptionkey@0"
I20260929 02:50:51.121409 17515 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 02:50:51.125455 17530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:51.126876 17515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:50:51.127107 17515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "9491ed1e1d32432cabf7cd7df81bdebd"
format_stamp: "Formatted at 2026-09-29 02:50:51 on dist-test-slave-6tjh"
server_key: "c79a9f16623ebd840c7c0779e13165a4"
server_key_iv: "edc74c836840e9f42e49c77e267da376"
server_key_version: "encryptionkey@0"
I20260929 02:50:51.127331 17515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:51.147044 17515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:51.151577 17515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:51.151886 17515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:51.171348 17515 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37785
I20260929 02:50:51.171391 17581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37785 every 8 connection(s)
I20260929 02:50:51.172978 17515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:51.177243 17582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:51.183310 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 17515
I20260929 02:50:51.183522 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:50:51.184230 388 external_mini_cluster.cc:1428] Setting key edb0b53c481497ae26562d53cb1b4f8e
I20260929 02:50:51.190116 17582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd: Bootstrap starting.
I20260929 02:50:51.194676 17582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:51.196048 17582 log.cc:824] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:51.199890 17582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd: No bootstrap required, opened a new log
Sep 29 02:50:51 dist-test-slave-6tjh krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:51.210474 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:51.186404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47326 (local address 127.0.97.62:37785)
0929 02:50:51.187332 (+ 928us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:51.187344 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:51.187385 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:50:51.189202 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:51.189227 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:51.189663 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:51.190005 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:51.192079 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:51.193607 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:51.195669 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:51.196500 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:51.196796 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:51.201995 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:51.202004 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:51.202021 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:51.202073 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:51.205502 (+ 3429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:51.206438 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:51.206445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:51.206453 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:51.206630 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:51.207098 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:51.207106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:51.207110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:51.207998 (+ 888us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:51.208261 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:51.208815 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 02:50:51.209210 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":400,"threads_started":1}
I20260929 02:50:51.210343 17582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260929 02:50:51.210830 17582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:51.210979 17582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9491ed1e1d32432cabf7cd7df81bdebd, State: Initialized, Role: FOLLOWER
I20260929 02:50:51.211602 17582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260929 02:50:51.211853 17582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:51.212020 17582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:51.212229 17582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:51.214933 17582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260929 02:50:51.215811 17582 leader_election.cc:302] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9491ed1e1d32432cabf7cd7df81bdebd; no voters:
I20260929 02:50:51.216534 17582 leader_election.cc:288] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:50:51.216692 17587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:51.217497 17587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [term 1 LEADER]: Becoming Leader. State: Replica: 9491ed1e1d32432cabf7cd7df81bdebd, State: Running, Role: LEADER
I20260929 02:50:51.218075 17587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } }
I20260929 02:50:51.218914 17582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:51.221490 17589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9491ed1e1d32432cabf7cd7df81bdebd. Latest consensus state: current_term: 1 leader_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } } }
I20260929 02:50:51.222306 17589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:51.222191 17588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9491ed1e1d32432cabf7cd7df81bdebd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37785 } } }
I20260929 02:50:51.224407 17588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:51.229741 17596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:51.235713 17596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:51.245846 17596 catalog_manager.cc:1347] Generated new cluster ID: 3d6deba2b77e4c9d956ff24a9c6c1421
I20260929 02:50:51.246122 17596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:51.270046 17596 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:50:51.272171 17596 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:51.281750 17596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd: Generated new TSK 0
I20260929 02:50:51.282900 17596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:50:51.353674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:51.717411 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:51.717736 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:51.717909 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:51.729434 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:51.729604 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:51.729858 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:50:51.741611 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:51.742878 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:51.744185 17608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:51.757956 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:51.758049 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:51.758080 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:51.759104 17608 server_base.cc:1034] running on GCE node
I20260929 02:50:51.760051 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:51.761176 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:51.762411 17608 hybrid_clock.cc:648] HybridClock initialized: now 1790650251762334 us; error 65 us; skew 500 ppm
I20260929 02:50:51.762718 17608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:51 dist-test-slave-6tjh krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:51.766332 17608 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:50:51.767885 17608 webserver.cc:460] Webserver started at http://127.0.97.1:40017/ using document root <none> and password file <none>
I20260929 02:50:51.768642 17608 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:51.768792 17608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:51.769075 17608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:51.772150 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "07fe0189144941adb96b813dd9326818"
format_stamp: "Formatted at 2026-09-29 02:50:51 on dist-test-slave-6tjh"
server_key: "d0ffc73e113f37a5be5e76c66dfa2069"
server_key_iv: "d583d2e78c8c051ec959a9b6217198c0"
server_key_version: "encryptionkey@0"
I20260929 02:50:51.773061 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "07fe0189144941adb96b813dd9326818"
format_stamp: "Formatted at 2026-09-29 02:50:51 on dist-test-slave-6tjh"
server_key: "d0ffc73e113f37a5be5e76c66dfa2069"
server_key_iv: "d583d2e78c8c051ec959a9b6217198c0"
server_key_version: "encryptionkey@0"
I20260929 02:50:51.778398 17608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:50:51.782291 17623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:51.784262 17608 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 02:50:51.784466 17608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "07fe0189144941adb96b813dd9326818"
format_stamp: "Formatted at 2026-09-29 02:50:51 on dist-test-slave-6tjh"
server_key: "d0ffc73e113f37a5be5e76c66dfa2069"
server_key_iv: "d583d2e78c8c051ec959a9b6217198c0"
server_key_version: "encryptionkey@0"
I20260929 02:50:51.784773 17608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:51.798691 17608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:51.802863 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:51.803117 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:51.804006 17608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:51.805459 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:51.805531 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:51.805617 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:51.805665 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:51.839787 17608 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45817
I20260929 02:50:51.839834 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45817 every 8 connection(s)
I20260929 02:50:51.841219 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:51.851370 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 17608
I20260929 02:50:51.851634 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:50:51.852620 388 external_mini_cluster.cc:1428] Setting key fad5ed143b151d8f94745cec47d00a43
Sep 29 02:50:51 dist-test-slave-6tjh krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:51.865381 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:51.843904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57373 (local address 127.0.97.62:37785)
0929 02:50:51.844379 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:51.844386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:51.845453 (+ 1067us) server_negotiation.cc:407] Connection header received
0929 02:50:51.847345 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:51.847349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:51.847430 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:51.847535 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:51.849190 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:51.849976 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:51.852760 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:51.853210 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:51.853369 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:51.858266 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:51.858271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:51.858275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:51.858318 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:51.861212 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:51.862388 (+ 1176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:51.862393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:51.862395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:51.862531 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:51.863136 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:51.863140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:51.863142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:51.863429 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:51.863552 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:51.864518 (+ 966us) server_negotiation.cc:299] Negotiation successful
0929 02:50:51.864848 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":219,"threads_started":1}
I20260929 02:50:51.866166 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:51.844501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37785 (local address 127.0.97.1:57373)
0929 02:50:51.845274 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:51.845319 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:51.847052 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:51.847775 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:51.847784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:51.848312 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:51.848962 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:51.848981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:51.850189 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:51.850194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:51.852502 (+ 2308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:51.852525 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:51.853485 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:51.853494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:51.853790 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:51.854748 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:51.854781 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:51.857916 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:51.861412 (+ 3496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:51.861428 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:51.861460 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:51.862213 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:51.862668 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:51.862677 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:51.862684 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:51.862995 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:51.863576 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:51.863588 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:51.864072 (+ 484us) client_negotiation.cc:769] Sending connection context
0929 02:50:51.864331 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 02:50:51.864901 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":278,"threads_started":1}
I20260929 02:50:51.868028 17736 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37785
I20260929 02:50:51.868438 17736 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:51.869315 17736 heartbeater.cc:511] Master 127.0.97.62:37785 requested a full tablet report, sending...
I20260929 02:50:51.871809 17547 ts_manager.cc:194] Registered new tserver with Master: 07fe0189144941adb96b813dd9326818 (127.0.97.1:45817)
I20260929 02:50:51.873378 17547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:57373
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:50:51.912333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:52.250471 17742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:52.250819 17742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:52.251001 17742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:52.258582 17742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:52.258744 17742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:52.258932 17742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:50:52.269177 17742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:52.270555 17742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:52.271951 17742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:52.286453 17750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:52.286466 17748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:52.286589 17747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:52.287768 17742 server_base.cc:1034] running on GCE node
I20260929 02:50:52.288663 17742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:52.289837 17742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:52.291097 17742 hybrid_clock.cc:648] HybridClock initialized: now 1790650252291052 us; error 44 us; skew 500 ppm
I20260929 02:50:52.291597 17742 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:52 dist-test-slave-6tjh krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:52.295415 17742 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:50:52.297297 17742 webserver.cc:460] Webserver started at http://127.0.97.2:38405/ using document root <none> and password file <none>
I20260929 02:50:52.298038 17742 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:52.298151 17742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:52.298418 17742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:52.301719 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "3e3c160fa55a4da2a1397ed72913cec8"
format_stamp: "Formatted at 2026-09-29 02:50:52 on dist-test-slave-6tjh"
server_key: "df0b413ac4b4dce384e749d6d9f25ec5"
server_key_iv: "e4f70c55ef7b7a31a3855a7968b75d05"
server_key_version: "encryptionkey@0"
I20260929 02:50:52.302573 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3e3c160fa55a4da2a1397ed72913cec8"
format_stamp: "Formatted at 2026-09-29 02:50:52 on dist-test-slave-6tjh"
server_key: "df0b413ac4b4dce384e749d6d9f25ec5"
server_key_iv: "e4f70c55ef7b7a31a3855a7968b75d05"
server_key_version: "encryptionkey@0"
I20260929 02:50:52.308672 17742 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 02:50:52.312485 17757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:52.314100 17742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:50:52.314250 17742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "3e3c160fa55a4da2a1397ed72913cec8"
format_stamp: "Formatted at 2026-09-29 02:50:52 on dist-test-slave-6tjh"
server_key: "df0b413ac4b4dce384e749d6d9f25ec5"
server_key_iv: "e4f70c55ef7b7a31a3855a7968b75d05"
server_key_version: "encryptionkey@0"
I20260929 02:50:52.314481 17742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:52.334167 17742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:52.339087 17742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:52.339432 17742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:52.340611 17742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:52.342126 17742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:52.342233 17742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:52.342396 17742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:52.342468 17742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:52.378096 17742 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33183
I20260929 02:50:52.378129 17869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33183 every 8 connection(s)
I20260929 02:50:52.379429 17742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:52.387969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 17742
I20260929 02:50:52.388195 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:50:52.388708 388 external_mini_cluster.cc:1428] Setting key f5216b10ee9ef6c9aecd63fcf3d874ef
Sep 29 02:50:52 dist-test-slave-6tjh krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:52.404178 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:52.382288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59649 (local address 127.0.97.62:37785)
0929 02:50:52.382766 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:52.382770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:52.383834 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 02:50:52.385837 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:52.385842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:52.385948 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:52.386077 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:52.387650 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.388591 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.390882 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.391642 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.391808 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:52.395942 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:52.395962 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:52.395970 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:52.396024 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:52.400276 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:52.401485 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:52.401490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:52.401493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:52.401642 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:52.402279 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:52.402283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:52.402285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:52.402578 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:52.402696 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:52.403662 (+ 966us) server_negotiation.cc:299] Negotiation successful
0929 02:50:52.403872 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":262,"threads_started":1}
I20260929 02:50:52.405809 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:52.382882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37785 (local address 127.0.97.2:59649)
0929 02:50:52.383628 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:52.383678 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:52.385533 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:52.386331 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:52.386340 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:52.386872 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:52.387457 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:52.387474 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.388750 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.388754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:52.390707 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:52.390723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.391880 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.391885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:52.392109 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:52.392909 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:52.392938 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:52.395633 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:52.400506 (+ 4873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:52.400531 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:52.400573 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:52.401319 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:52.401777 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:52.401785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:52.401791 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:52.402054 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:52.402713 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:52.402725 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:52.403227 (+ 502us) client_negotiation.cc:769] Sending connection context
0929 02:50:52.403561 (+ 334us) client_negotiation.cc:240] Negotiation successful
0929 02:50:52.404204 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":285,"threads_started":1}
I20260929 02:50:52.407657 17870 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37785
I20260929 02:50:52.408061 17870 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:52.408975 17870 heartbeater.cc:511] Master 127.0.97.62:37785 requested a full tablet report, sending...
I20260929 02:50:52.410849 17547 ts_manager.cc:194] Registered new tserver with Master: 3e3c160fa55a4da2a1397ed72913cec8 (127.0.97.2:33183)
I20260929 02:50:52.412199 17547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59649
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:50:52.453372 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:52.797849 17876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:52.798207 17876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:52.798388 17876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:52.805970 17876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:52.806161 17876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:52.806354 17876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:50:52.816859 17876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37785
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:52.818441 17876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:52.820437 17876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:52.836155 17881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:52.836071 17882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:52.836117 17884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:52.837435 17876 server_base.cc:1034] running on GCE node
I20260929 02:50:52.838109 17876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:52.839236 17876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:52.840600 17876 hybrid_clock.cc:648] HybridClock initialized: now 1790650252840482 us; error 67 us; skew 500 ppm
I20260929 02:50:52.840950 17876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:52 dist-test-slave-6tjh krb5kdc[17502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:52.844470 17876 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:50:52.845966 17876 webserver.cc:460] Webserver started at http://127.0.97.3:38043/ using document root <none> and password file <none>
I20260929 02:50:52.846614 17876 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:52.846712 17876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:52.846953 17876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:52.849920 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "f167ad4f43a54421ade57c9df70a55f4"
format_stamp: "Formatted at 2026-09-29 02:50:52 on dist-test-slave-6tjh"
server_key: "ba72959c8b815d07e8e3f2d23fdbf23a"
server_key_iv: "3d0ba9c9fc8f42b7e01085a89d4f6536"
server_key_version: "encryptionkey@0"
I20260929 02:50:52.850689 17876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f167ad4f43a54421ade57c9df70a55f4"
format_stamp: "Formatted at 2026-09-29 02:50:52 on dist-test-slave-6tjh"
server_key: "ba72959c8b815d07e8e3f2d23fdbf23a"
server_key_iv: "3d0ba9c9fc8f42b7e01085a89d4f6536"
server_key_version: "encryptionkey@0"
I20260929 02:50:52.856504 17876 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:50:52.860154 17891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:52.862027 17876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:50:52.862274 17876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "f167ad4f43a54421ade57c9df70a55f4"
format_stamp: "Formatted at 2026-09-29 02:50:52 on dist-test-slave-6tjh"
server_key: "ba72959c8b815d07e8e3f2d23fdbf23a"
server_key_iv: "3d0ba9c9fc8f42b7e01085a89d4f6536"
server_key_version: "encryptionkey@0"
I20260929 02:50:52.862535 17876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:52.876767 17736 heartbeater.cc:503] Master 127.0.97.62:37785 was elected leader, sending a full tablet report...
I20260929 02:50:52.885761 17876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:52.890650 17876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:52.890971 17876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:52.892096 17876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:52.893707 17876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:52.893818 17876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:52.893983 17876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:52.894064 17876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:52.929629 17876 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46557
I20260929 02:50:52.929678 18003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46557 every 8 connection(s)
I20260929 02:50:52.931285 17876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:52.939550 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 17876
I20260929 02:50:52.939893 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:50:52.940444 388 external_mini_cluster.cc:1428] Setting key 9058bfb6a1ab772dc2c9d8f815f1d810
Sep 29 02:50:52 dist-test-slave-6tjh krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650252, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:52.952875 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:52.934271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36631 (local address 127.0.97.62:37785)
0929 02:50:52.934745 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:52.934751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:52.935853 (+ 1102us) server_negotiation.cc:407] Connection header received
0929 02:50:52.937763 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:52.937768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:52.937847 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:52.937949 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:52.939598 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.940785 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.942445 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.942874 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.943005 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:52.946753 (+ 3748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:52.946758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:52.946762 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:52.946794 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:52.949899 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:52.950682 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:52.950686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:52.950689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:52.950817 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:52.951213 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:52.951216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:52.951218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:52.951499 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:52.951619 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:52.952343 (+ 724us) server_negotiation.cc:299] Negotiation successful
0929 02:50:52.952528 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":215,"threads_started":1}
I20260929 02:50:52.953912 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:52.934866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37785 (local address 127.0.97.3:36631)
0929 02:50:52.935692 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:52.935735 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:52.937495 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:52.938218 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:52.938227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:52.938784 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:52.939381 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:52.939399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.940959 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.940970 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:52.942280 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:52.942293 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.943120 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.943134 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:52.943463 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:52.944159 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:52.944182 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:52.946535 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:52.950076 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:52.950083 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:52.950100 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:52.950561 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:52.950917 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:52.950921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:52.950923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:52.951118 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:52.951621 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:52.951627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:52.951965 (+ 338us) client_negotiation.cc:769] Sending connection context
0929 02:50:52.952220 (+ 255us) client_negotiation.cc:240] Negotiation successful
0929 02:50:52.952748 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":291,"threads_started":1}
I20260929 02:50:52.955457 18004 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37785
I20260929 02:50:52.955775 18004 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:52.956660 18004 heartbeater.cc:511] Master 127.0.97.62:37785 requested a full tablet report, sending...
I20260929 02:50:52.958253 17546 ts_manager.cc:194] Registered new tserver with Master: f167ad4f43a54421ade57c9df70a55f4 (127.0.97.3:46557)
I20260929 02:50:52.959125 17546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36631
I20260929 02:50:52.969236 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:50:52.987911 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:52.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.0.97.62:37785)
0929 02:50:52.974393 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:52.974400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:52.974667 (+ 267us) server_negotiation.cc:407] Connection header received
0929 02:50:52.974815 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:52.974820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:52.974893 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:52.975027 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:52.976708 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.977535 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.978976 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:52.979479 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:52.979603 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:52.982313 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:52.982318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:52.982322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:52.982359 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:52.985047 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:52.985804 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:52.985809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:52.985812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:52.985966 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:52.986337 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:52.986340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:52.986342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:52.986705 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:52.986901 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:52.987326 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 02:50:52.987533 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260929 02:50:52.993028 17546 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:50:53.000034 17546 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:50:53.035295 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.013365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.0.97.3:46557)
0929 02:50:53.025114 (+ 11749us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.025123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.025835 (+ 712us) server_negotiation.cc:407] Connection header received
0929 02:50:53.026187 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.026198 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.026499 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.026712 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:53.027611 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.029475 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.032391 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.033992 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.034132 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.034381 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.034711 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.034909 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11552,"thread_start_us":11750,"threads_started":1}
I20260929 02:50:53.035374 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.013082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46557 (local address 127.0.0.1:38504)
0929 02:50:53.025711 (+ 12629us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:53.025731 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:53.025970 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:53.026812 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:53.026816 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:53.026854 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:53.027354 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.027366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.029684 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.029688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.032115 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.032151 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.034202 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.034209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.034430 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.034492 (+ 62us) client_negotiation.cc:769] Sending connection context
0929 02:50:53.034649 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 02:50:53.034871 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12495,"mutex_wait_us":4,"spinlock_wait_cycles":8320,"thread_start_us":12351,"threads_started":1}
I20260929 02:50:53.035456 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.012953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45817 (local address 127.0.0.1:41234)
0929 02:50:53.025215 (+ 12262us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:53.025276 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:53.025655 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:53.026800 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:53.026807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:53.026855 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:53.027708 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.027717 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.030219 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.030223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.032373 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.032388 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.034547 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.034552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.034729 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.034760 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 02:50:53.034848 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:50:53.034974 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11874,"mutex_wait_us":317,"spinlock_wait_cycles":5120,"thread_start_us":11667,"threads_started":1}
I20260929 02:50:53.036381 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.013211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.0.97.1:45817)
0929 02:50:53.024605 (+ 11394us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.024620 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.025454 (+ 834us) server_negotiation.cc:407] Connection header received
0929 02:50:53.025948 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.025958 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.026247 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.026419 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:53.027885 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.030049 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.032598 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.033992 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.034132 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.034381 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.035842 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.036036 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11105,"thread_start_us":10894,"threads_started":1}
I20260929 02:50:53.037056 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.027038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:33183 (local address 127.0.0.1:46270)
0929 02:50:53.027997 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:53.028016 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:53.028148 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:53.029120 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:53.029125 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:53.029152 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:53.029591 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.029603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.031864 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.031869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.034049 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.034068 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.036279 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.036285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.036438 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.036468 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:50:53.036582 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 02:50:53.036724 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":809,"threads_started":1}
I20260929 02:50:53.037770 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.025998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.97.2:33183)
0929 02:50:53.027407 (+ 1409us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.027419 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.028137 (+ 718us) server_negotiation.cc:407] Connection header received
0929 02:50:53.028348 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.028356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.028619 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.028813 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:53.029886 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.031685 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.034258 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.035801 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.035923 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.036133 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.037249 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.037427 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":1158,"threads_started":1}
I20260929 02:50:53.039999 17939 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebe794abcaa479ab4e2f538f69f7393 (DEFAULT_TABLE table=test-table [id=0ad9ebf87dde4179bf2ff7f769f39b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:53.039999 17671 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebe794abcaa479ab4e2f538f69f7393 (DEFAULT_TABLE table=test-table [id=0ad9ebf87dde4179bf2ff7f769f39b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:53.041600 17939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebe794abcaa479ab4e2f538f69f7393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:53.041600 17671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebe794abcaa479ab4e2f538f69f7393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:53.043995 17805 tablet_service.cc:1467] Processing CreateTablet for tablet 3ebe794abcaa479ab4e2f538f69f7393 (DEFAULT_TABLE table=test-table [id=0ad9ebf87dde4179bf2ff7f769f39b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:53.045665 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ebe794abcaa479ab4e2f538f69f7393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:53.057082 18023 tablet_bootstrap.cc:492] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Bootstrap starting.
I20260929 02:50:53.060036 18024 tablet_bootstrap.cc:492] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Bootstrap starting.
I20260929 02:50:53.062109 18025 tablet_bootstrap.cc:492] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Bootstrap starting.
I20260929 02:50:53.062145 18023 tablet_bootstrap.cc:654] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:53.063537 18023 log.cc:824] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:53.063921 18024 tablet_bootstrap.cc:654] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:53.065250 18024 log.cc:824] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:53.065807 18025 tablet_bootstrap.cc:654] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:53.067307 18025 log.cc:824] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:53.068849 18023 tablet_bootstrap.cc:492] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: No bootstrap required, opened a new log
I20260929 02:50:53.069262 18023 ts_tablet_manager.cc:1397] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 02:50:53.075197 18024 tablet_bootstrap.cc:492] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: No bootstrap required, opened a new log
I20260929 02:50:53.075464 18025 tablet_bootstrap.cc:492] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: No bootstrap required, opened a new log
I20260929 02:50:53.075505 18024 ts_tablet_manager.cc:1397] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20260929 02:50:53.075730 18025 ts_tablet_manager.cc:1397] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260929 02:50:53.080837 18023 raft_consensus.cc:348] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.081310 18023 raft_consensus.cc:374] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:53.081465 18023 raft_consensus.cc:729] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3c160fa55a4da2a1397ed72913cec8, State: Initialized, Role: FOLLOWER
I20260929 02:50:53.082154 18023 consensus_queue.cc:260] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.083330 17870 heartbeater.cc:503] Master 127.0.97.62:37785 was elected leader, sending a full tablet report...
I20260929 02:50:53.084002 18023 ts_tablet_manager.cc:1428] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260929 02:50:53.084452 18024 raft_consensus.cc:348] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.084920 18024 raft_consensus.cc:374] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:53.085080 18024 raft_consensus.cc:729] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07fe0189144941adb96b813dd9326818, State: Initialized, Role: FOLLOWER
I20260929 02:50:53.085704 18024 consensus_queue.cc:260] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.086208 18025 raft_consensus.cc:348] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.086724 18025 raft_consensus.cc:374] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:53.086863 18025 raft_consensus.cc:729] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f167ad4f43a54421ade57c9df70a55f4, State: Initialized, Role: FOLLOWER
I20260929 02:50:53.087462 18024 ts_tablet_manager.cc:1428] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260929 02:50:53.087648 18025 consensus_queue.cc:260] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.093478 18004 heartbeater.cc:503] Master 127.0.97.62:37785 was elected leader, sending a full tablet report...
W20260929 02:50:53.093717 17737 tablet.cc:2369] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:53.094298 18025 ts_tablet_manager.cc:1428] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Time spent starting tablet: real 0.018s user 0.009s sys 0.010s
W20260929 02:50:53.131767 17871 tablet.cc:2369] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:50:53.183348 18005 tablet.cc:2369] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:53.257205 18029 raft_consensus.cc:482] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:50:53.257495 18029 raft_consensus.cc:504] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.259212 18029 leader_election.cc:288] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07fe0189144941adb96b813dd9326818 (127.0.97.1:45817), f167ad4f43a54421ade57c9df70a55f4 (127.0.97.3:46557)
I20260929 02:50:53.265911 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.259656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58297 (local address 127.0.97.1:45817)
0929 02:50:53.259892 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.259901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.260417 (+ 516us) server_negotiation.cc:407] Connection header received
0929 02:50:53.260684 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.260688 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.260749 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.260855 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:53.261466 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.262190 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.264090 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.264965 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.265114 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.265228 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.265527 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.265651 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:50:53.265938 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.259680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44095 (local address 127.0.97.3:46557)
0929 02:50:53.259890 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.259898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.260514 (+ 616us) server_negotiation.cc:407] Connection header received
0929 02:50:53.260764 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.260769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.260828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.260931 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:53.261584 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.262296 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.264085 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.264935 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.265076 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.265192 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.265511 (+ 319us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.265636 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 02:50:53.266173 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.259472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45817 (local address 127.0.97.2:58297)
0929 02:50:53.260281 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:53.260300 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:53.260491 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:53.260866 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:53.260871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:53.260893 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:53.261310 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.261320 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.262321 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.262329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.263931 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.263943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.265199 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.265205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.265372 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.265393 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 02:50:53.265500 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 02:50:53.265624 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":48,"spinlock_wait_cycles":18304,"thread_start_us":538,"threads_started":1}
I20260929 02:50:53.266134 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.259545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46557 (local address 127.0.97.2:44095)
0929 02:50:53.260405 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:53.260420 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:53.260588 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:53.260956 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:53.260960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:53.260999 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:53.261412 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.261421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.262438 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.262444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.263931 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:53.263943 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.265174 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.265184 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:53.265344 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.265371 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 02:50:53.265477 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 02:50:53.265608 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":30,"thread_start_us":355,"threads_started":1}
I20260929 02:50:53.267458 17959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebe794abcaa479ab4e2f538f69f7393" candidate_uuid: "3e3c160fa55a4da2a1397ed72913cec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f167ad4f43a54421ade57c9df70a55f4" is_pre_election: true
I20260929 02:50:53.267480 17691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebe794abcaa479ab4e2f538f69f7393" candidate_uuid: "3e3c160fa55a4da2a1397ed72913cec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fe0189144941adb96b813dd9326818" is_pre_election: true
I20260929 02:50:53.268059 17959 raft_consensus.cc:2454] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e3c160fa55a4da2a1397ed72913cec8 in term 0.
I20260929 02:50:53.268065 17691 raft_consensus.cc:2454] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e3c160fa55a4da2a1397ed72913cec8 in term 0.
I20260929 02:50:53.268777 17758 leader_election.cc:302] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07fe0189144941adb96b813dd9326818, 3e3c160fa55a4da2a1397ed72913cec8, f167ad4f43a54421ade57c9df70a55f4; no voters:
I20260929 02:50:53.269273 18029 raft_consensus.cc:2781] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:50:53.269433 18029 raft_consensus.cc:482] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:50:53.269536 18029 raft_consensus.cc:3037] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:53.271783 18029 raft_consensus.cc:504] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.272588 18029 leader_election.cc:288] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [CANDIDATE]: Term 1 election: Requested vote from peers 07fe0189144941adb96b813dd9326818 (127.0.97.1:45817), f167ad4f43a54421ade57c9df70a55f4 (127.0.97.3:46557)
I20260929 02:50:53.273030 17691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebe794abcaa479ab4e2f538f69f7393" candidate_uuid: "3e3c160fa55a4da2a1397ed72913cec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07fe0189144941adb96b813dd9326818"
I20260929 02:50:53.273132 17959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ebe794abcaa479ab4e2f538f69f7393" candidate_uuid: "3e3c160fa55a4da2a1397ed72913cec8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f167ad4f43a54421ade57c9df70a55f4"
I20260929 02:50:53.273253 17691 raft_consensus.cc:3037] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:53.273384 17959 raft_consensus.cc:3037] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:53.276451 17691 raft_consensus.cc:2454] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e3c160fa55a4da2a1397ed72913cec8 in term 1.
I20260929 02:50:53.276693 17959 raft_consensus.cc:2454] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e3c160fa55a4da2a1397ed72913cec8 in term 1.
I20260929 02:50:53.276952 17758 leader_election.cc:302] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07fe0189144941adb96b813dd9326818, 3e3c160fa55a4da2a1397ed72913cec8; no voters:
I20260929 02:50:53.277254 18029 raft_consensus.cc:2781] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:53.277779 18029 raft_consensus.cc:686] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 1 LEADER]: Becoming Leader. State: Replica: 3e3c160fa55a4da2a1397ed72913cec8, State: Running, Role: LEADER
I20260929 02:50:53.278280 18029 consensus_queue.cc:237] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } }
I20260929 02:50:53.283493 17546 catalog_manager.cc:5495] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e3c160fa55a4da2a1397ed72913cec8 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "3e3c160fa55a4da2a1397ed72913cec8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e3c160fa55a4da2a1397ed72913cec8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 33183 } health_report { overall_health: HEALTHY } } }
I20260929 02:50:53.339525 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.333308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.0.97.2:33183)
0929 02:50:53.333579 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.333585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.333609 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:50:53.333720 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.333724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.333805 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.333978 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:53.334398 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.335199 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.336971 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.337543 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.337693 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.338288 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:53.338835 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:53.339029 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.339088 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.339181 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 02:50:53.348628 17691 raft_consensus.cc:1261] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 1 FOLLOWER]: Refusing update from remote peer 3e3c160fa55a4da2a1397ed72913cec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:53.348629 17959 raft_consensus.cc:1261] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 1 FOLLOWER]: Refusing update from remote peer 3e3c160fa55a4da2a1397ed72913cec8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:53.350062 18034 consensus_queue.cc:1035] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07fe0189144941adb96b813dd9326818" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:53.350513 18029 consensus_queue.cc:1035] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f167ad4f43a54421ade57c9df70a55f4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:53.365276 18039 mvcc.cc:205] Tried to move back new op lower bound from 7334503437704343552 to 7334503437431508992. Current Snapshot: MvccSnapshot[applied={T|T < 7334503437704343552}]
I20260929 02:50:53.369208 18038 mvcc.cc:205] Tried to move back new op lower bound from 7334503437704343552 to 7334503437431508992. Current Snapshot: MvccSnapshot[applied={T|T < 7334503437704343552 or (T in {7334503437704343552})}]
I20260929 02:50:53.378439 17547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:53.378743 17547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:53.382428 17547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9491ed1e1d32432cabf7cd7df81bdebd: Sending DeleteTablet for 3 replicas of tablet 3ebe794abcaa479ab4e2f538f69f7393
I20260929 02:50:53.383708 17671 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ebe794abcaa479ab4e2f538f69f7393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41234
I20260929 02:50:53.384295 17939 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ebe794abcaa479ab4e2f538f69f7393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:38504
I20260929 02:50:53.387420 17805 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ebe794abcaa479ab4e2f538f69f7393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:53 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46270
I20260929 02:50:53.394776 18047 tablet_replica.cc:330] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: stopping tablet replica
I20260929 02:50:53.396018 18047 raft_consensus.cc:2229] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:53.396684 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 17608
I20260929 02:50:53.398416 18047 raft_consensus.cc:2258] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:53.402000 18047 ts_tablet_manager.cc:1905] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:53.403286 18048 tablet_replica.cc:330] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: stopping tablet replica
I20260929 02:50:53.403857 18048 raft_consensus.cc:2229] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:50:53.404634 18048 raft_consensus.cc:2258] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:53.411079 18046 tablet_replica.cc:330] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: stopping tablet replica
I20260929 02:50:53.411540 18046 raft_consensus.cc:2229] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:53.412094 18046 raft_consensus.cc:2258] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:53.412575 18048 ts_tablet_manager.cc:1905] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:53.412653 18047 ts_tablet_manager.cc:1918] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:53.412892 18047 log.cc:1196] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/3ebe794abcaa479ab4e2f538f69f7393
I20260929 02:50:53.413491 18047 ts_tablet_manager.cc:1939] T 3ebe794abcaa479ab4e2f538f69f7393 P f167ad4f43a54421ade57c9df70a55f4: Deleting consensus metadata
I20260929 02:50:53.415565 17533 catalog_manager.cc:4841] TS f167ad4f43a54421ade57c9df70a55f4 (127.0.97.3:46557): tablet 3ebe794abcaa479ab4e2f538f69f7393 (table test-table [id=0ad9ebf87dde4179bf2ff7f769f39b1a]) successfully deleted
Sep 29 02:50:53 dist-test-slave-6tjh krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:50:53.419109 18046 ts_tablet_manager.cc:1905] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:53.421154 18048 ts_tablet_manager.cc:1918] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:53.421365 18048 log.cc:1196] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/3ebe794abcaa479ab4e2f538f69f7393
I20260929 02:50:53.421984 18048 ts_tablet_manager.cc:1939] T 3ebe794abcaa479ab4e2f538f69f7393 P 3e3c160fa55a4da2a1397ed72913cec8: Deleting consensus metadata
I20260929 02:50:53.424098 17531 catalog_manager.cc:4841] TS 3e3c160fa55a4da2a1397ed72913cec8 (127.0.97.2:33183): tablet 3ebe794abcaa479ab4e2f538f69f7393 (table test-table [id=0ad9ebf87dde4179bf2ff7f769f39b1a]) successfully deleted
I20260929 02:50:53.424837 18046 ts_tablet_manager.cc:1918] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:53.425199 18046 log.cc:1196] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/3ebe794abcaa479ab4e2f538f69f7393
I20260929 02:50:53.425849 18046 ts_tablet_manager.cc:1939] T 3ebe794abcaa479ab4e2f538f69f7393 P 07fe0189144941adb96b813dd9326818: Deleting consensus metadata
I20260929 02:50:53.427572 17531 catalog_manager.cc:4841] TS 07fe0189144941adb96b813dd9326818 (127.0.97.1:45817): tablet 3ebe794abcaa479ab4e2f538f69f7393 (table test-table [id=0ad9ebf87dde4179bf2ff7f769f39b1a]) successfully deleted
I20260929 02:50:53.433430 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.397581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.0.97.1:45817)
0929 02:50:53.397809 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.397820 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.401931 (+ 4111us) server_negotiation.cc:407] Connection header received
0929 02:50:53.405151 (+ 3220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.405159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.405282 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.405365 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:53.407145 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.408287 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.412520 (+ 4233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.413306 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.413417 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.425410 (+ 11993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:53.425428 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:53.425449 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:53.425506 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:53.430008 (+ 4502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:53.430805 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:53.430817 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:53.430821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:53.431004 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:53.431468 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:53.431474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:53.431478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:53.431993 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:53.432338 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.432740 (+ 402us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.432994 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:50:53.434130 17731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:53.525418 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 17608
I20260929 02:50:53.550011 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 17742
Sep 29 02:50:53 dist-test-slave-6tjh krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:50:53.566468 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.550848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46286 (local address 127.0.97.2:33183)
0929 02:50:53.551049 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.551055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.551075 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:50:53.551267 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.551272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.551359 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.551468 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:53.552884 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.553738 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.555091 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.555530 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.555667 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.560043 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:53.560059 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:53.560074 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:53.560121 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:53.563315 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:53.564153 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:53.564164 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:53.564166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:53.564317 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:53.564756 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:53.564761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:53.564765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:53.565142 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:53.565515 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.565815 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.566046 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:50:53.567224 17865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:53.681082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 17742
I20260929 02:50:53.699213 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 17876
Sep 29 02:50:53 dist-test-slave-6tjh krb5kdc[17502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:50:53.716418 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:53.700004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.0.97.3:46557)
0929 02:50:53.700208 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:53.700216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:53.700237 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:50:53.700418 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:53.700430 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:53.700496 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:53.700680 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:53.702004 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.702751 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.704733 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:53.705134 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:53.705277 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:53.710270 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:53.710288 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:53.710306 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:53.710349 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:53.713458 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:53.714392 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:53.714402 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:53.714404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:53.714601 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:53.715042 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:53.715046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:53.715048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:53.715428 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:53.715683 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:53.715987 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 02:50:53.716157 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:50:53.717166 17999 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:53.810001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 17876
I20260929 02:50:53.826833 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 17515
I20260929 02:50:53.827998 17577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:53.941504 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 17515
2026-09-29T02:50:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4622 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T02:50:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:54Z Disabled control of system clock
I20260929 02:50:54.025972 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41987
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:50:54.365341 18057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:54.365669 18057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:54.365752 18057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:54.373271 18057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:54.373400 18057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:54.373440 18057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:54.373489 18057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:54.373528 18057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:54.382844 18057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:54.384703 18057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:54.386475 18057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:54.398732 18062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:54.398645 18063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:54.398685 18065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:54.399595 18057 server_base.cc:1034] running on GCE node
I20260929 02:50:54.400471 18057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:54.402076 18057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:54.403316 18057 hybrid_clock.cc:648] HybridClock initialized: now 1790650254403258 us; error 56 us; skew 500 ppm
I20260929 02:50:54.403707 18057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:54.411394 18057 webserver.cc:460] Webserver started at http://127.0.97.62:42687/ using document root <none> and password file <none>
I20260929 02:50:54.412181 18057 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:54.412323 18057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:54.412724 18057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:54.416059 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "a9f6cdc8fba446ed847276920e2f969c"
format_stamp: "Formatted at 2026-09-29 02:50:54 on dist-test-slave-6tjh"
server_key: "43967d4726b8e16499fa66138473dbef"
server_key_iv: "db54c17f79ffc09c4afb94ecf00acaad"
server_key_version: "encryptionkey@0"
I20260929 02:50:54.417055 18057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "a9f6cdc8fba446ed847276920e2f969c"
format_stamp: "Formatted at 2026-09-29 02:50:54 on dist-test-slave-6tjh"
server_key: "43967d4726b8e16499fa66138473dbef"
server_key_iv: "db54c17f79ffc09c4afb94ecf00acaad"
server_key_version: "encryptionkey@0"
I20260929 02:50:54.422811 18057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:50:54.426769 18071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:54.428357 18057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:50:54.428633 18057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "a9f6cdc8fba446ed847276920e2f969c"
format_stamp: "Formatted at 2026-09-29 02:50:54 on dist-test-slave-6tjh"
server_key: "43967d4726b8e16499fa66138473dbef"
server_key_iv: "db54c17f79ffc09c4afb94ecf00acaad"
server_key_version: "encryptionkey@0"
I20260929 02:50:54.428885 18057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:54.441351 18057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:54.442230 18057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:54.442533 18057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:54.462070 18057 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41987
I20260929 02:50:54.462096 18122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41987 every 8 connection(s)
I20260929 02:50:54.463510 18057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:54.467548 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:54.471191 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18057
I20260929 02:50:54.471397 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:50:54.471964 388 external_mini_cluster.cc:1428] Setting key 69bc576d0c92cb4eb3d04c39ae59f1c5
I20260929 02:50:54.480671 18123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c: Bootstrap starting.
I20260929 02:50:54.483784 18123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:54.484947 18123 log.cc:824] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:54.487819 18123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c: No bootstrap required, opened a new log
I20260929 02:50:54.491667 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:54.474204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.0.97.62:41987)
0929 02:50:54.475245 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:54.475257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:54.475301 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:50:54.477108 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:54.477137 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:54.477699 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:54.478159 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:54.478826 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:54.480087 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:54.481594 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:54.482315 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:54.482537 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:54.489053 (+ 6516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:54.489063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:54.489078 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:54.489750 (+ 672us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:54.489838 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:54.489866 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:54.490020 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:54.490293 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 02:50:54.490660 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":329,"threads_started":1}
I20260929 02:50:54.496511 18123 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f6cdc8fba446ed847276920e2f969c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 02:50:54.496985 18123 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:54.497105 18123 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9f6cdc8fba446ed847276920e2f969c, State: Initialized, Role: FOLLOWER
I20260929 02:50:54.497725 18123 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f6cdc8fba446ed847276920e2f969c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 02:50:54.497977 18123 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:54.498121 18123 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:54.498306 18123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:54.501029 18123 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f6cdc8fba446ed847276920e2f969c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 02:50:54.501565 18123 leader_election.cc:302] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9f6cdc8fba446ed847276920e2f969c; no voters:
I20260929 02:50:54.502180 18123 leader_election.cc:288] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:50:54.502461 18128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:54.503224 18128 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [term 1 LEADER]: Becoming Leader. State: Replica: a9f6cdc8fba446ed847276920e2f969c, State: Running, Role: LEADER
I20260929 02:50:54.503731 18128 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f6cdc8fba446ed847276920e2f969c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } }
I20260929 02:50:54.504340 18123 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:54.506978 18130 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9f6cdc8fba446ed847276920e2f969c. Latest consensus state: current_term: 1 leader_uuid: "a9f6cdc8fba446ed847276920e2f969c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f6cdc8fba446ed847276920e2f969c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } } }
I20260929 02:50:54.507452 18130 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:54.508088 18129 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9f6cdc8fba446ed847276920e2f969c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9f6cdc8fba446ed847276920e2f969c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41987 } } }
I20260929 02:50:54.508383 18129 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:54.510586 18137 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:54.516206 18137 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:54.525954 18137 catalog_manager.cc:1347] Generated new cluster ID: 9c3570759f3549f0b44e9c7ce4bdf3d5
I20260929 02:50:54.526125 18137 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:54.540304 18137 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:50:54.542341 18137 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:54.557453 18137 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c: Generated new TSK 0
I20260929 02:50:54.558641 18137 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:50:54.568935 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41987
--builtin_ntp_servers=127.0.97.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:50:54.941833 18147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:54.942235 18147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:54.942415 18147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:54.949811 18147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:54.950007 18147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:54.950206 18147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:50:54.959637 18147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:54.961051 18147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:54.962815 18147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:54.975919 18152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:54.975953 18153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:54.976188 18155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:54.976876 18147 server_base.cc:1034] running on GCE node
I20260929 02:50:54.977631 18147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:54.978657 18147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:54.979907 18147 hybrid_clock.cc:648] HybridClock initialized: now 1790650254979856 us; error 45 us; skew 500 ppm
I20260929 02:50:54.980285 18147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:54.982721 18147 webserver.cc:460] Webserver started at http://127.0.97.1:43795/ using document root <none> and password file <none>
I20260929 02:50:54.983458 18147 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:54.983592 18147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:54.983918 18147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:54.987093 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "45e048e7f1d44f1f88fa341a86a910d0"
format_stamp: "Formatted at 2026-09-29 02:50:54 on dist-test-slave-6tjh"
server_key: "341f837c73598165fb3cb22e6d8e5330"
server_key_iv: "32837df9e9cfadae3883b9eafcc093fa"
server_key_version: "encryptionkey@0"
I20260929 02:50:54.987953 18147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "45e048e7f1d44f1f88fa341a86a910d0"
format_stamp: "Formatted at 2026-09-29 02:50:54 on dist-test-slave-6tjh"
server_key: "341f837c73598165fb3cb22e6d8e5330"
server_key_iv: "32837df9e9cfadae3883b9eafcc093fa"
server_key_version: "encryptionkey@0"
I20260929 02:50:54.993839 18147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260929 02:50:54.997675 18161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:54.999255 18147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:50:54.999469 18147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "45e048e7f1d44f1f88fa341a86a910d0"
format_stamp: "Formatted at 2026-09-29 02:50:54 on dist-test-slave-6tjh"
server_key: "341f837c73598165fb3cb22e6d8e5330"
server_key_iv: "32837df9e9cfadae3883b9eafcc093fa"
server_key_version: "encryptionkey@0"
I20260929 02:50:54.999716 18147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:55.061518 18147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:55.062445 18147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:55.062738 18147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:55.063757 18147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:55.065497 18147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:55.065619 18147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:55.065757 18147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:55.065843 18147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:55.100947 18147 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42229
I20260929 02:50:55.100982 18273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42229 every 8 connection(s)
I20260929 02:50:55.102355 18147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:55.106078 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18147
I20260929 02:50:55.106271 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:50:55.106737 388 external_mini_cluster.cc:1428] Setting key 1e35a9565973ab4fd116980447a4791a
I20260929 02:50:55.110041 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41987
--builtin_ntp_servers=127.0.97.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:50:55.117491 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:55.105172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46235 (local address 127.0.97.62:41987)
0929 02:50:55.105679 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:55.105705 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:55.107106 (+ 1401us) server_negotiation.cc:407] Connection header received
0929 02:50:55.109595 (+ 2489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:55.109603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:55.109710 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:55.109844 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:55.111052 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.112244 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.113920 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.114667 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.114822 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:55.116434 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:55.116443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:55.116451 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:55.116479 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:55.116559 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:55.116579 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:55.116675 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:55.116911 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 02:50:55.117035 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":278,"threads_started":1}
I20260929 02:50:55.118613 18277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:55.105788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41987 (local address 127.0.97.1:46235)
0929 02:50:55.106885 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:55.106949 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:55.109282 (+ 2333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:55.110078 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:55.110087 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:55.110214 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:55.110861 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:55.110881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.112404 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.112408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:55.113761 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:55.113773 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.114934 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.114940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:55.115232 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:55.116071 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:55.116074 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:55.116118 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:55.116202 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:55.116693 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:55.116702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:55.116709 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:50:55.116866 (+ 157us) client_negotiation.cc:240] Negotiation successful
0929 02:50:55.117199 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":349,"threads_started":1}
I20260929 02:50:55.120299 18274 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41987
I20260929 02:50:55.120723 18274 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:55.121537 18274 heartbeater.cc:511] Master 127.0.97.62:41987 requested a full tablet report, sending...
I20260929 02:50:55.123690 18088 ts_manager.cc:194] Registered new tserver with Master: 45e048e7f1d44f1f88fa341a86a910d0 (127.0.97.1:42229)
I20260929 02:50:55.125351 18088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:46235
W20260929 02:50:55.447048 18278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:55.447342 18278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:55.447480 18278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:55.454907 18278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:55.455026 18278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:55.455199 18278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:50:55.464617 18278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:55.466032 18278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:55.467857 18278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:55.481911 18283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:55.481896 18284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:55.482102 18286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:55.482415 18278 server_base.cc:1034] running on GCE node
I20260929 02:50:55.483448 18278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:55.484485 18278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:55.485771 18278 hybrid_clock.cc:648] HybridClock initialized: now 1790650255485725 us; error 50 us; skew 500 ppm
I20260929 02:50:55.486122 18278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:55.488840 18278 webserver.cc:460] Webserver started at http://127.0.97.2:37041/ using document root <none> and password file <none>
I20260929 02:50:55.489635 18278 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:55.489802 18278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:55.490165 18278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:55.493451 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "fd68c757d77c4ac0a2352af2e878726f"
format_stamp: "Formatted at 2026-09-29 02:50:55 on dist-test-slave-6tjh"
server_key: "17e9e55a88c5571a257482ab384cfebf"
server_key_iv: "d976cacdc65a1bd37faad10d109cd744"
server_key_version: "encryptionkey@0"
I20260929 02:50:55.494346 18278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "fd68c757d77c4ac0a2352af2e878726f"
format_stamp: "Formatted at 2026-09-29 02:50:55 on dist-test-slave-6tjh"
server_key: "17e9e55a88c5571a257482ab384cfebf"
server_key_iv: "d976cacdc65a1bd37faad10d109cd744"
server_key_version: "encryptionkey@0"
I20260929 02:50:55.499964 18278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:50:55.503836 18292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:55.505494 18278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:50:55.505671 18278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "fd68c757d77c4ac0a2352af2e878726f"
format_stamp: "Formatted at 2026-09-29 02:50:55 on dist-test-slave-6tjh"
server_key: "17e9e55a88c5571a257482ab384cfebf"
server_key_iv: "d976cacdc65a1bd37faad10d109cd744"
server_key_version: "encryptionkey@0"
I20260929 02:50:55.505959 18278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:55.528367 18278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:55.529403 18278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:55.529757 18278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:55.530855 18278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:55.532605 18278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:55.532740 18278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 02:50:55.532886 18278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:55.532984 18278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:55.569300 18278 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41783
I20260929 02:50:55.569406 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41783 every 8 connection(s)
I20260929 02:50:55.570750 18278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:55.577934 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18278
I20260929 02:50:55.578135 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:50:55.578727 388 external_mini_cluster.cc:1428] Setting key 3dc3cf70a2ef7d300f5ea8811266d495
I20260929 02:50:55.582046 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41987
--builtin_ntp_servers=127.0.97.20:42885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:50:55.586035 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:55.573665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45459 (local address 127.0.97.62:41987)
0929 02:50:55.573879 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:55.573884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:55.575414 (+ 1530us) server_negotiation.cc:407] Connection header received
0929 02:50:55.577323 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:55.577328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:55.577402 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:55.577509 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:55.579064 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.579997 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.582336 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.583112 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.583250 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:55.585249 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:55.585255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:55.585258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:55.585277 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:55.585311 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:55.585324 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:55.585404 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:55.585678 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 02:50:55.585764 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:50:55.587535 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:55.574251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41987 (local address 127.0.97.2:45459)
0929 02:50:55.575241 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:55.575285 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:55.577037 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:55.577808 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:55.577821 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:55.577997 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:55.578825 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:55.578852 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.580167 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.580173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:55.582152 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:55.582170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:55.583614 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:55.583623 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:55.583961 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:55.584856 (+ 895us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:55.584862 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:55.584918 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:55.585035 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:55.585438 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:55.585451 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:55.585461 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:50:55.585643 (+ 182us) client_negotiation.cc:240] Negotiation successful
0929 02:50:55.586082 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":316,"threads_started":1}
I20260929 02:50:55.589270 18405 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41987
I20260929 02:50:55.589614 18405 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:55.590571 18405 heartbeater.cc:511] Master 127.0.97.62:41987 requested a full tablet report, sending...
I20260929 02:50:55.592101 18088 ts_manager.cc:194] Registered new tserver with Master: fd68c757d77c4ac0a2352af2e878726f (127.0.97.2:41783)
I20260929 02:50:55.592979 18088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:45459
W20260929 02:50:55.923385 18408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:55.923702 18408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:55.923852 18408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:55.931476 18408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:55.931597 18408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:55.931779 18408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:50:55.941423 18408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41987
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:55.942770 18408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:55.944880 18408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:55.958668 18413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:55.958647 18416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:55.958581 18414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:55.959793 18408 server_base.cc:1034] running on GCE node
I20260929 02:50:55.960371 18408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:55.961429 18408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:55.962651 18408 hybrid_clock.cc:648] HybridClock initialized: now 1790650255962602 us; error 39 us; skew 500 ppm
I20260929 02:50:55.962951 18408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:50:55.965452 18408 webserver.cc:460] Webserver started at http://127.0.97.3:44261/ using document root <none> and password file <none>
I20260929 02:50:55.966107 18408 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:55.966202 18408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:55.966459 18408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:55.969363 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "96a1225b91a54b599a57ecdc4adb313d"
format_stamp: "Formatted at 2026-09-29 02:50:55 on dist-test-slave-6tjh"
server_key: "3ea529c6b4d3d9ce45fd8df4848fdfc1"
server_key_iv: "8f124e3fd3db4493ab549688ec88e736"
server_key_version: "encryptionkey@0"
I20260929 02:50:55.970157 18408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "96a1225b91a54b599a57ecdc4adb313d"
format_stamp: "Formatted at 2026-09-29 02:50:55 on dist-test-slave-6tjh"
server_key: "3ea529c6b4d3d9ce45fd8df4848fdfc1"
server_key_iv: "8f124e3fd3db4493ab549688ec88e736"
server_key_version: "encryptionkey@0"
I20260929 02:50:55.975512 18408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:50:55.979331 18422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:55.981015 18408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:50:55.981168 18408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "96a1225b91a54b599a57ecdc4adb313d"
format_stamp: "Formatted at 2026-09-29 02:50:55 on dist-test-slave-6tjh"
server_key: "3ea529c6b4d3d9ce45fd8df4848fdfc1"
server_key_iv: "8f124e3fd3db4493ab549688ec88e736"
server_key_version: "encryptionkey@0"
I20260929 02:50:55.981366 18408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:56.011804 18408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:56.012753 18408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:56.013041 18408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:56.014104 18408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:56.015695 18408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:56.015820 18408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:56.015973 18408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:56.016062 18408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:56.049785 18408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43855
I20260929 02:50:56.049820 18534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43855 every 8 connection(s)
I20260929 02:50:56.051204 18408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:50:56.060210 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18408
I20260929 02:50:56.060422 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:50:56.060950 388 external_mini_cluster.cc:1428] Setting key 148f03ec9ef9f3e46fd7a7deaea5f5eb
I20260929 02:50:56.065213 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.054008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39563 (local address 127.0.97.62:41987)
0929 02:50:56.054228 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.054233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.055747 (+ 1514us) server_negotiation.cc:407] Connection header received
0929 02:50:56.057753 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.057758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.057853 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.058014 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:56.059202 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.060154 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.062213 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.062729 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.062850 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.064375 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:56.064381 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:56.064384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:56.064402 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:56.064436 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:56.064448 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:56.064519 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.064845 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.064934 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:50:56.066320 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.054577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41987 (local address 127.0.97.3:39563)
0929 02:50:56.055567 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.055605 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:56.057413 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:56.058254 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:56.058263 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:56.058368 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:56.058965 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.058987 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.060341 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.060345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.062015 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.062049 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.062972 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.062982 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.063250 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.064042 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:50:56.064045 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:56.064098 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:50:56.064188 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:56.064539 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:56.064600 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:56.064614 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 02:50:56.064784 (+ 170us) client_negotiation.cc:240] Negotiation successful
0929 02:50:56.065158 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":322,"threads_started":1}
I20260929 02:50:56.067806 18535 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41987
I20260929 02:50:56.068127 18535 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:56.068964 18535 heartbeater.cc:511] Master 127.0.97.62:41987 requested a full tablet report, sending...
I20260929 02:50:56.070485 18088 ts_manager.cc:194] Registered new tserver with Master: 96a1225b91a54b599a57ecdc4adb313d (127.0.97.3:43855)
I20260929 02:50:56.071350 18088 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39563
I20260929 02:50:56.076741 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:50:56.089167 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.082810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.0.97.62:41987)
0929 02:50:56.082987 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.082992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.083431 (+ 439us) server_negotiation.cc:407] Connection header received
0929 02:50:56.083638 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.083643 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.083714 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.083816 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:56.084313 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.085183 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.086692 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.087134 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.087243 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.088347 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:56.088352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:56.088355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:56.088375 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:56.088407 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:56.088419 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:56.088531 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.088804 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.088893 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:50:56.093432 18088 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:50:56.100476 18088 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:50:56.126724 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.117423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43456)
0929 02:50:56.118039 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.118059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:56.118168 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:56.119730 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:56.119737 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:56.119767 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:56.120115 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.120126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.121962 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.121969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.123673 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.123685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.125635 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.125642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.125867 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.125924 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 02:50:56.126091 (+ 167us) client_negotiation.cc:240] Negotiation successful
0929 02:50:56.126317 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"mutex_wait_us":138,"thread_start_us":942,"threads_started":1}
I20260929 02:50:56.127085 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.115592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.0.97.3:43855)
0929 02:50:56.116450 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.116461 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.117209 (+ 748us) server_negotiation.cc:407] Connection header received
0929 02:50:56.117633 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.117641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.117966 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.118170 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:56.119160 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.120776 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.123819 (+ 3043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.125427 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.125572 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.125865 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.126508 (+ 643us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.126733 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":470,"threads_started":1}
I20260929 02:50:56.127302 18547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.115346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43855 (local address 127.0.0.1:48022)
0929 02:50:56.117050 (+ 1704us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.117066 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:56.117298 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:56.118563 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:56.118570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:56.118617 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:56.118949 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.118962 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.120948 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.120955 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.123571 (+ 2616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.123589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.126019 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.126024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.126242 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.126283 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 02:50:56.126419 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 02:50:56.126571 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1331,"mutex_wait_us":106,"thread_start_us":1102,"threads_started":1}
I20260929 02:50:56.127406 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.116887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.0.97.2:41783)
0929 02:50:56.118675 (+ 1788us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.118685 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.118719 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:50:56.118812 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.118819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.119088 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.119303 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:56.119863 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.121527 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.124232 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.125986 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.126119 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.126348 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.126841 (+ 493us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.127044 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1611,"thread_start_us":1482,"threads_started":1}
I20260929 02:50:56.127010 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.116235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41783 (local address 127.0.0.1:51408)
0929 02:50:56.117009 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.117058 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:56.117290 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:56.119285 (+ 1995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:56.119289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:56.119315 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:56.119628 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.119663 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.121672 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.121677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.124004 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.124017 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.126181 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.126188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.126362 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.126407 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:50:56.126484 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 02:50:56.126591 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":87,"thread_start_us":826,"threads_started":1}
I20260929 02:50:56.128232 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.118121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.0.97.1:42229)
0929 02:50:56.119108 (+ 987us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.119118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.119147 (+ 29us) server_negotiation.cc:407] Connection header received
0929 02:50:56.119252 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.119261 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.119497 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.119720 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:56.120323 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.121824 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.123930 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.125427 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.125572 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.125865 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.127687 (+ 1822us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.127879 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":622,"threads_started":1}
I20260929 02:50:56.129163 18274 heartbeater.cc:503] Master 127.0.97.62:41987 was elected leader, sending a full tablet report...
I20260929 02:50:56.131827 18209 tablet_service.cc:1467] Processing CreateTablet for tablet 11e46140eb804f9da7193fc51968ad08 (DEFAULT_TABLE table=test-table [id=5bbde10ca69146889394294144ebdc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:56.131862 18470 tablet_service.cc:1467] Processing CreateTablet for tablet 11e46140eb804f9da7193fc51968ad08 (DEFAULT_TABLE table=test-table [id=5bbde10ca69146889394294144ebdc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:56.131862 18340 tablet_service.cc:1467] Processing CreateTablet for tablet 11e46140eb804f9da7193fc51968ad08 (DEFAULT_TABLE table=test-table [id=5bbde10ca69146889394294144ebdc33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:50:56.133302 18470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e46140eb804f9da7193fc51968ad08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:56.133302 18340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e46140eb804f9da7193fc51968ad08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:56.133306 18209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e46140eb804f9da7193fc51968ad08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:56.145484 18553 tablet_bootstrap.cc:492] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: Bootstrap starting.
I20260929 02:50:56.149873 18553 tablet_bootstrap.cc:654] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:56.151000 18553 log.cc:824] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:56.152086 18554 tablet_bootstrap.cc:492] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Bootstrap starting.
I20260929 02:50:56.153132 18555 tablet_bootstrap.cc:492] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Bootstrap starting.
I20260929 02:50:56.157706 18554 tablet_bootstrap.cc:654] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:56.158914 18555 tablet_bootstrap.cc:654] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:56.159005 18554 log.cc:824] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:56.159746 18553 tablet_bootstrap.cc:492] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: No bootstrap required, opened a new log
I20260929 02:50:56.160032 18553 ts_tablet_manager.cc:1397] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260929 02:50:56.160149 18555 log.cc:824] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:50:56.162272 18554 tablet_bootstrap.cc:492] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: No bootstrap required, opened a new log
I20260929 02:50:56.162556 18554 ts_tablet_manager.cc:1397] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 02:50:56.162878 18555 tablet_bootstrap.cc:492] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: No bootstrap required, opened a new log
I20260929 02:50:56.163142 18555 ts_tablet_manager.cc:1397] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20260929 02:50:56.170800 18555 raft_consensus.cc:348] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.170842 18553 raft_consensus.cc:348] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.171249 18555 raft_consensus.cc:374] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:56.171339 18553 raft_consensus.cc:374] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:56.171368 18555 raft_consensus.cc:729] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a1225b91a54b599a57ecdc4adb313d, State: Initialized, Role: FOLLOWER
I20260929 02:50:56.171442 18553 raft_consensus.cc:729] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45e048e7f1d44f1f88fa341a86a910d0, State: Initialized, Role: FOLLOWER
I20260929 02:50:56.172111 18555 consensus_queue.cc:260] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.172248 18553 consensus_queue.cc:260] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.173327 18535 heartbeater.cc:503] Master 127.0.97.62:41987 was elected leader, sending a full tablet report...
I20260929 02:50:56.173964 18554 raft_consensus.cc:348] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.174527 18554 raft_consensus.cc:374] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:56.174669 18554 raft_consensus.cc:729] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd68c757d77c4ac0a2352af2e878726f, State: Initialized, Role: FOLLOWER
I20260929 02:50:56.174758 18555 ts_tablet_manager.cc:1428] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260929 02:50:56.175499 18554 consensus_queue.cc:260] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.177292 18405 heartbeater.cc:503] Master 127.0.97.62:41987 was elected leader, sending a full tablet report...
I20260929 02:50:56.177592 18553 ts_tablet_manager.cc:1428] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260929 02:50:56.179616 18554 ts_tablet_manager.cc:1428] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260929 02:50:56.259831 18560 raft_consensus.cc:482] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:50:56.260154 18560 raft_consensus.cc:504] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.261916 18560 leader_election.cc:288] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96a1225b91a54b599a57ecdc4adb313d (127.0.97.3:43855), fd68c757d77c4ac0a2352af2e878726f (127.0.97.2:41783)
I20260929 02:50:56.272233 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.262524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:45971 (local address 127.0.97.3:43855)
0929 02:50:56.262761 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.262769 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.263292 (+ 523us) server_negotiation.cc:407] Connection header received
0929 02:50:56.263526 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.263532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.263623 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.263791 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:56.264821 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.266119 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.269452 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.270941 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.271132 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.271293 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.271763 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.271924 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:50:56.272256 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.262499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43467 (local address 127.0.97.2:41783)
0929 02:50:56.262733 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.262742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.263411 (+ 669us) server_negotiation.cc:407] Connection header received
0929 02:50:56.263538 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.263544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.263648 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.263791 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:50:56.264821 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.266119 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.269452 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.270979 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.271130 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.271249 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.271767 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.271958 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:50:56.272333 18563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.262277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43855 (local address 127.0.97.1:45971)
0929 02:50:56.263162 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.263182 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:56.263358 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:56.263825 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:56.263830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:56.263871 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:56.264583 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.264599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.266299 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.266304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.269158 (+ 2854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.269176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.271261 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.271267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.271468 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.271578 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 02:50:56.271685 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 02:50:56.271864 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":257,"spinlock_wait_cycles":94464,"thread_start_us":697,"threads_started":1}
I20260929 02:50:56.272294 18562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.262277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41783 (local address 127.0.97.1:43467)
0929 02:50:56.263298 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.263319 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:56.263421 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:56.263825 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:56.263830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:56.263858 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:50:56.264583 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.264599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.266298 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.266304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.269161 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:56.269176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.271248 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.271256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:56.271470 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.271583 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 02:50:56.271686 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 02:50:56.271864 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":423,"thread_start_us":392,"threads_started":1}
I20260929 02:50:56.273478 18360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e46140eb804f9da7193fc51968ad08" candidate_uuid: "45e048e7f1d44f1f88fa341a86a910d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd68c757d77c4ac0a2352af2e878726f" is_pre_election: true
I20260929 02:50:56.274003 18360 raft_consensus.cc:2454] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45e048e7f1d44f1f88fa341a86a910d0 in term 0.
I20260929 02:50:56.273989 18490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e46140eb804f9da7193fc51968ad08" candidate_uuid: "45e048e7f1d44f1f88fa341a86a910d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a1225b91a54b599a57ecdc4adb313d" is_pre_election: true
I20260929 02:50:56.274638 18490 raft_consensus.cc:2454] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45e048e7f1d44f1f88fa341a86a910d0 in term 0.
I20260929 02:50:56.274718 18162 leader_election.cc:302] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e048e7f1d44f1f88fa341a86a910d0, fd68c757d77c4ac0a2352af2e878726f; no voters:
I20260929 02:50:56.275218 18560 raft_consensus.cc:2781] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:50:56.275436 18560 raft_consensus.cc:482] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:50:56.275583 18560 raft_consensus.cc:3037] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:56.278473 18560 raft_consensus.cc:504] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.279279 18560 leader_election.cc:288] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [CANDIDATE]: Term 1 election: Requested vote from peers 96a1225b91a54b599a57ecdc4adb313d (127.0.97.3:43855), fd68c757d77c4ac0a2352af2e878726f (127.0.97.2:41783)
I20260929 02:50:56.279831 18490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e46140eb804f9da7193fc51968ad08" candidate_uuid: "45e048e7f1d44f1f88fa341a86a910d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a1225b91a54b599a57ecdc4adb313d"
I20260929 02:50:56.280076 18360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11e46140eb804f9da7193fc51968ad08" candidate_uuid: "45e048e7f1d44f1f88fa341a86a910d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd68c757d77c4ac0a2352af2e878726f"
I20260929 02:50:56.280138 18490 raft_consensus.cc:3037] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:56.280308 18360 raft_consensus.cc:3037] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:56.282923 18360 raft_consensus.cc:2454] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e048e7f1d44f1f88fa341a86a910d0 in term 1.
I20260929 02:50:56.282930 18490 raft_consensus.cc:2454] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45e048e7f1d44f1f88fa341a86a910d0 in term 1.
I20260929 02:50:56.283506 18164 leader_election.cc:302] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45e048e7f1d44f1f88fa341a86a910d0, 96a1225b91a54b599a57ecdc4adb313d; no voters:
I20260929 02:50:56.283905 18560 raft_consensus.cc:2781] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:56.284390 18560 raft_consensus.cc:686] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 1 LEADER]: Becoming Leader. State: Replica: 45e048e7f1d44f1f88fa341a86a910d0, State: Running, Role: LEADER
I20260929 02:50:56.285111 18560 consensus_queue.cc:237] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } }
I20260929 02:50:56.290351 18086 catalog_manager.cc:5495] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45e048e7f1d44f1f88fa341a86a910d0 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "45e048e7f1d44f1f88fa341a86a910d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45e048e7f1d44f1f88fa341a86a910d0" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 } health_report { overall_health: UNKNOWN } } }
W20260929 02:50:56.303143 18536 tablet.cc:2369] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:50:56.323040 18406 tablet.cc:2369] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:56.330569 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.324735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.0.97.1:42229)
0929 02:50:56.324914 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.324918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.324936 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:50:56.325074 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.325077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.325164 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.325289 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:50:56.325610 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.326450 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.328232 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.328770 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.328910 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.329432 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:50:56.329970 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:50:56.330142 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.330195 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.330288 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:50:56.339735 18360 raft_consensus.cc:1261] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 1 FOLLOWER]: Refusing update from remote peer 45e048e7f1d44f1f88fa341a86a910d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:56.339735 18490 raft_consensus.cc:1261] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 1 FOLLOWER]: Refusing update from remote peer 45e048e7f1d44f1f88fa341a86a910d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:50:56.341043 18564 consensus_queue.cc:1035] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a1225b91a54b599a57ecdc4adb313d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:50:56.341454 18560 consensus_queue.cc:1035] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd68c757d77c4ac0a2352af2e878726f" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:50:56.354889 18275 tablet.cc:2369] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:50:56.356148 18567 mvcc.cc:205] Tried to move back new op lower bound from 7334503449956003840 to 7334503449748701184. Current Snapshot: MvccSnapshot[applied={T|T < 7334503449956003840}]
I20260929 02:50:56.372927 18086 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:56.373262 18086 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:50:56.377594 18086 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9f6cdc8fba446ed847276920e2f969c: Sending DeleteTablet for 3 replicas of tablet 11e46140eb804f9da7193fc51968ad08
I20260929 02:50:56.379168 18340 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e46140eb804f9da7193fc51968ad08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:56 UTC) from {username='slave'} at 127.0.0.1:51408
I20260929 02:50:56.379144 18209 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e46140eb804f9da7193fc51968ad08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:56 UTC) from {username='slave'} at 127.0.0.1:43456
I20260929 02:50:56.379419 18470 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e46140eb804f9da7193fc51968ad08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:50:56 UTC) from {username='slave'} at 127.0.0.1:48022
I20260929 02:50:56.381054 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18147
I20260929 02:50:56.389526 18579 tablet_replica.cc:330] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: stopping tablet replica
I20260929 02:50:56.391179 18579 raft_consensus.cc:2229] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:56.392885 18579 raft_consensus.cc:2258] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:56.396942 18579 ts_tablet_manager.cc:1905] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:56.403635 18577 tablet_replica.cc:330] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: stopping tablet replica
I20260929 02:50:56.404433 18579 ts_tablet_manager.cc:1918] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:56.404495 18577 raft_consensus.cc:2229] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:50:56.406255 18577 raft_consensus.cc:2258] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:56.409119 18579 log.cc:1196] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/11e46140eb804f9da7193fc51968ad08
I20260929 02:50:56.409471 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.383615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.0.97.1:42229)
0929 02:50:56.383887 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.383897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.395927 (+ 12030us) server_negotiation.cc:407] Connection header received
0929 02:50:56.396604 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.396611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.396745 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.396873 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:56.397985 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.399806 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.403947 (+ 4141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.404773 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.404946 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.407100 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:56.407116 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:56.407136 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:56.408087 (+ 951us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:56.408157 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:56.408190 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:56.408310 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.408619 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.408754 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:50:56.409761 18579 ts_tablet_manager.cc:1939] T 11e46140eb804f9da7193fc51968ad08 P 96a1225b91a54b599a57ecdc4adb313d: Deleting consensus metadata
I20260929 02:50:56.410070 18578 tablet_replica.cc:330] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: stopping tablet replica
I20260929 02:50:56.411224 18269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:56.411229 18578 raft_consensus.cc:2229] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:50:56.411882 18578 raft_consensus.cc:2258] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:50:56.411851 18074 catalog_manager.cc:4841] TS 96a1225b91a54b599a57ecdc4adb313d (127.0.97.3:43855): tablet 11e46140eb804f9da7193fc51968ad08 (table test-table [id=5bbde10ca69146889394294144ebdc33]) successfully deleted
I20260929 02:50:56.414873 18578 ts_tablet_manager.cc:1905] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:56.422317 18578 ts_tablet_manager.cc:1918] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:50:56.422484 18578 log.cc:1196] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/11e46140eb804f9da7193fc51968ad08
I20260929 02:50:56.423022 18578 ts_tablet_manager.cc:1939] T 11e46140eb804f9da7193fc51968ad08 P fd68c757d77c4ac0a2352af2e878726f: Deleting consensus metadata
I20260929 02:50:56.425035 18072 catalog_manager.cc:4841] TS fd68c757d77c4ac0a2352af2e878726f (127.0.97.2:41783): tablet 11e46140eb804f9da7193fc51968ad08 (table test-table [id=5bbde10ca69146889394294144ebdc33]) successfully deleted
I20260929 02:50:56.411881 18577 ts_tablet_manager.cc:1905] T 11e46140eb804f9da7193fc51968ad08 P 45e048e7f1d44f1f88fa341a86a910d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:50:56.514251 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18147
W20260929 02:50:56.533808 18074 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:42229 (error 108)
I20260929 02:50:56.534006 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18278
I20260929 02:50:56.535728 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.535317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43488)
0929 02:50:56.535482 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.535593 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:50:56.536015 18074 catalog_manager.cc:4584] TS 45e048e7f1d44f1f88fa341a86a910d0 (127.0.97.1:42229): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 11e46140eb804f9da7193fc51968ad08: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
I20260929 02:50:56.541168 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.534854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.0.97.2:41783)
0929 02:50:56.535070 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.535078 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.535100 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:50:56.535297 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.535301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.535380 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.535511 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:56.535977 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.536828 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.538071 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.538461 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.538555 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.539617 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:56.539627 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:56.539642 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:56.540250 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:56.540297 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:56.540317 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:56.540419 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.540613 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.540750 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:50:56.541965 18400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:56.589804 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.589107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43496)
0929 02:50:56.589347 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.589491 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 02:50:56.590979 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.590563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43504)
0929 02:50:56.590733 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.590823 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:50:56.634290 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18278
I20260929 02:50:56.651839 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.651364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43516)
0929 02:50:56.651584 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.651683 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:50:56.652570 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18408
I20260929 02:50:56.653266 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.652783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43532)
0929 02:50:56.653000 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.653106 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 02:50:56.660283 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.653602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.0.97.3:43855)
0929 02:50:56.653793 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:56.653798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:56.653817 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:50:56.653873 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:56.653876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:56.653960 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:56.654052 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:56.654564 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.655507 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.657246 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:56.657664 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:56.657812 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:56.658904 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:56.658913 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:56.658929 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:50:56.659512 (+ 583us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:56.659566 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:50:56.659589 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:56.659725 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:56.659844 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 02:50:56.659961 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 02:50:56.661108 18530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:56.734901 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.734355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43540)
0929 02:50:56.734591 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.734718 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 02:50:56.736054 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:56.735637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42229 (local address 127.0.0.1:43550)
0929 02:50:56.735836 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:56.735939 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:42229: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260929 02:50:56.751216 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18408
I20260929 02:50:56.768441 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18057
I20260929 02:50:56.769723 18118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:50:56.880729 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18057
2026-09-29T02:50:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2931 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:50:56 dist-test-slave-6tjh krb5kdc[18588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:50:56 dist-test-slave-6tjh krb5kdc[18588](info): set up 2 sockets
Sep 29 02:50:56 dist-test-slave-6tjh krb5kdc[18588](info): commencing operation
krb5kdc: starting...
W20260929 02:50:58.044818 388 mini_kdc.cc:121] Time spent starting KDC: real 1.122s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:50:58 dist-test-slave-6tjh krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:50:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:50:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:50:58.317826 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38833
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:58.671912 18601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:58.672266 18601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:58.672360 18601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:58.679836 18601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:50:58.679976 18601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:58.680024 18601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:58.680073 18601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:50:58.680111 18601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:50:58.690759 18601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:58.692142 18601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:58.693604 18601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:58.705703 18607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:58.705737 18606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:58.706041 18609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:58.706881 18601 server_base.cc:1034] running on GCE node
I20260929 02:50:58.707691 18601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:58.709303 18601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:58.710512 18601 hybrid_clock.cc:648] HybridClock initialized: now 1790650258710472 us; error 40 us; skew 500 ppm
I20260929 02:50:58.710906 18601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:58 dist-test-slave-6tjh krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:58.714215 18601 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:50:58.715756 18601 webserver.cc:460] Webserver started at http://127.0.97.62:34447/ using document root <none> and password file <none>
I20260929 02:50:58.716475 18601 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:58.716660 18601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:58.717001 18601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:58.720054 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26"
format_stamp: "Formatted at 2026-09-29 02:50:58 on dist-test-slave-6tjh"
server_key: "8050e68080333a0378499c4c87effeaf"
server_key_iv: "8488c59cd3c492465f8858cfead98244"
server_key_version: "encryptionkey@0"
I20260929 02:50:58.720875 18601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26"
format_stamp: "Formatted at 2026-09-29 02:50:58 on dist-test-slave-6tjh"
server_key: "8050e68080333a0378499c4c87effeaf"
server_key_iv: "8488c59cd3c492465f8858cfead98244"
server_key_version: "encryptionkey@0"
I20260929 02:50:58.726599 18601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:50:58.730592 18616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:58.732084 18601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:50:58.732295 18601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26"
format_stamp: "Formatted at 2026-09-29 02:50:58 on dist-test-slave-6tjh"
server_key: "8050e68080333a0378499c4c87effeaf"
server_key_iv: "8488c59cd3c492465f8858cfead98244"
server_key_version: "encryptionkey@0"
I20260929 02:50:58.732499 18601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:58.751238 18601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:58.755827 18601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:58.756137 18601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:58.775583 18601 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38833
I20260929 02:50:58.775615 18667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38833 every 8 connection(s)
I20260929 02:50:58.777092 18601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:50:58.780956 18668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:50:58.784425 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18601
I20260929 02:50:58.784642 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:50:58.785259 388 external_mini_cluster.cc:1428] Setting key aa7accaaaa1910295263b666adc5d485
I20260929 02:50:58.794468 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26: Bootstrap starting.
I20260929 02:50:58.798799 18668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26: Neither blocks nor log segments found. Creating new log.
I20260929 02:50:58.800271 18668 log.cc:824] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:50:58 dist-test-slave-6tjh krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:50:58.803683 18668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26: No bootstrap required, opened a new log
I20260929 02:50:58.810643 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:58.787353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.0.97.62:38833)
0929 02:50:58.788148 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:58.788160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:58.788204 (+ 44us) server_negotiation.cc:407] Connection header received
0929 02:50:58.790011 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:58.790036 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:58.790490 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:58.790862 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:58.792717 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:58.794072 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:58.796167 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:58.797033 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:58.797198 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:58.802544 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:58.802552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:58.802568 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:58.802610 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:58.805958 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:58.806935 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:58.806942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:58.806949 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:58.807171 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:58.807624 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:58.807628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:58.807629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:58.808224 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:58.808462 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:58.809000 (+ 538us) server_negotiation.cc:299] Negotiation successful
0929 02:50:58.809475 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":357,"threads_started":1}
I20260929 02:50:58.813452 18668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38833 } }
I20260929 02:50:58.813938 18668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:50:58.814100 18668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1abcb477a64fc1a19f1dfbdba6fc26, State: Initialized, Role: FOLLOWER
I20260929 02:50:58.814715 18668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38833 } }
I20260929 02:50:58.814968 18668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:50:58.815122 18668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:50:58.815308 18668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:50:58.817672 18668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38833 } }
I20260929 02:50:58.818213 18668 leader_election.cc:302] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b1abcb477a64fc1a19f1dfbdba6fc26; no voters:
I20260929 02:50:58.818837 18668 leader_election.cc:288] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:50:58.819100 18673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:50:58.820324 18673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [term 1 LEADER]: Becoming Leader. State: Replica: 1b1abcb477a64fc1a19f1dfbdba6fc26, State: Running, Role: LEADER
I20260929 02:50:58.820910 18673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38833 } }
I20260929 02:50:58.821103 18668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:50:58.824290 18674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38833 } } }
I20260929 02:50:58.824450 18675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b1abcb477a64fc1a19f1dfbdba6fc26. Latest consensus state: current_term: 1 leader_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b1abcb477a64fc1a19f1dfbdba6fc26" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38833 } } }
I20260929 02:50:58.824832 18674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:58.824842 18675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26 [sys.catalog]: This master's current role is: LEADER
I20260929 02:50:58.825714 18682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:50:58.832347 18682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:50:58.842111 18682 catalog_manager.cc:1347] Generated new cluster ID: 5a5202b181f4476e9b751fd4b2b120ea
I20260929 02:50:58.842253 18682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:50:58.888688 18682 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:50:58.890084 18682 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:50:58.901157 18682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26: Generated new TSK 0
I20260929 02:50:58.902220 18682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:50:58.964370 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38833
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:59.313853 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:59.314148 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:59.314311 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:59.322041 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:59.322192 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:59.322378 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:50:59.332979 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:59.334362 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:59.335774 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:59.350360 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:59.350261 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:59.350242 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:59.352212 18694 server_base.cc:1034] running on GCE node
I20260929 02:50:59.352946 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:59.354053 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:59.355314 18694 hybrid_clock.cc:648] HybridClock initialized: now 1790650259355257 us; error 52 us; skew 500 ppm
I20260929 02:50:59.355713 18694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:59 dist-test-slave-6tjh krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:59.359318 18694 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:50:59.361078 18694 webserver.cc:460] Webserver started at http://127.0.97.1:44539/ using document root <none> and password file <none>
I20260929 02:50:59.361843 18694 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:59.362004 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:59.362362 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:59.365839 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "2bb069f608b94d4da0871b507a602fb6"
format_stamp: "Formatted at 2026-09-29 02:50:59 on dist-test-slave-6tjh"
server_key: "1b0babc65f3b89bfd3d0372579fb3f3f"
server_key_iv: "3730b565a7e66061c96865f9acfee01d"
server_key_version: "encryptionkey@0"
I20260929 02:50:59.366726 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2bb069f608b94d4da0871b507a602fb6"
format_stamp: "Formatted at 2026-09-29 02:50:59 on dist-test-slave-6tjh"
server_key: "1b0babc65f3b89bfd3d0372579fb3f3f"
server_key_iv: "3730b565a7e66061c96865f9acfee01d"
server_key_version: "encryptionkey@0"
I20260929 02:50:59.372480 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:50:59.376986 18709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:59.378530 18694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:50:59.378731 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "2bb069f608b94d4da0871b507a602fb6"
format_stamp: "Formatted at 2026-09-29 02:50:59 on dist-test-slave-6tjh"
server_key: "1b0babc65f3b89bfd3d0372579fb3f3f"
server_key_iv: "3730b565a7e66061c96865f9acfee01d"
server_key_version: "encryptionkey@0"
I20260929 02:50:59.378966 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:59.392534 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:59.397601 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:59.397938 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:59.398991 18694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:59.400694 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:59.400805 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:59.400965 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:59.401053 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:59.435907 18694 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39279
I20260929 02:50:59.435925 18821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39279 every 8 connection(s)
I20260929 02:50:59.437453 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:50:59.439602 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18694
I20260929 02:50:59.439791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:50:59.440397 388 external_mini_cluster.cc:1428] Setting key 312181ec7511a395f9fa1d0f53d11515
Sep 29 02:50:59 dist-test-slave-6tjh krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260929 02:50:59.471407 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:59.440540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:52869 (local address 127.0.97.62:38833)
0929 02:50:59.441095 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:59.441100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:59.442153 (+ 1053us) server_negotiation.cc:407] Connection header received
0929 02:50:59.445009 (+ 2856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:59.445013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:59.445086 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:59.445188 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:59.447495 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:59.448245 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:59.450961 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:50:59.451392 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:50:59.451510 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:59.463448 (+ 11938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:50:59.463460 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:50:59.463469 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:50:59.463528 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:50:59.467696 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:59.468830 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:59.468835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:59.468838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:59.468978 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:50:59.469504 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:50:59.469508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:50:59.469510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:50:59.469812 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:50:59.469995 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:50:59.470780 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 02:50:59.471032 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":264,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260929 02:50:59.472882 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:59.441231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38833 (local address 127.0.97.1:52869)
0929 02:50:59.441992 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:59.442039 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:59.444658 (+ 2619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:59.445490 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:59.445505 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:59.446369 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:59.447280 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:59.447308 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:59.448423 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:59.448431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:59.450735 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:59.450757 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:50:59.451663 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:50:59.451673 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:50:59.452022 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:50:59.460055 (+ 8033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:50:59.460086 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:50:59.462905 (+ 2819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:50:59.468012 (+ 5107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:59.468023 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:59.468041 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:59.468652 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:59.469114 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:50:59.469119 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:50:59.469123 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:50:59.469347 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:50:59.469959 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:50:59.469979 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:50:59.470376 (+ 397us) client_negotiation.cc:769] Sending connection context
0929 02:50:59.470657 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 02:50:59.471280 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":341,"threads_started":1}
I20260929 02:50:59.474887 18822 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38833
I20260929 02:50:59.475253 18822 heartbeater.cc:464] Registering TS with master...
I20260929 02:50:59.476205 18822 heartbeater.cc:511] Master 127.0.97.62:38833 requested a full tablet report, sending...
I20260929 02:50:59.478853 18633 ts_manager.cc:194] Registered new tserver with Master: 2bb069f608b94d4da0871b507a602fb6 (127.0.97.1:39279)
I20260929 02:50:59.480933 18633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:52869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:50:59.508074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38833
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:50:59.860246 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:50:59.860599 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:50:59.860707 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:50:59.868393 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:50:59.868572 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:50:59.868702 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:50:59.880672 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:50:59.882055 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:50:59.884085 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:50:59.898296 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:59.898383 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:50:59.898279 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:50:59.899019 18828 server_base.cc:1034] running on GCE node
I20260929 02:50:59.899748 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:50:59.900861 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:50:59.902179 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790650259902101 us; error 75 us; skew 500 ppm
I20260929 02:50:59.902565 18828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:50:59 dist-test-slave-6tjh krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:50:59.906020 18828 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:50:59.907554 18828 webserver.cc:460] Webserver started at http://127.0.97.2:39097/ using document root <none> and password file <none>
I20260929 02:50:59.908279 18828 fs_manager.cc:359] Metadata directory not provided
I20260929 02:50:59.908386 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:50:59.908741 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:50:59.911870 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "ef079377d96e443e8f785d975f7074f5"
format_stamp: "Formatted at 2026-09-29 02:50:59 on dist-test-slave-6tjh"
server_key: "a867650a370742dc2ed13cb2a51b0bdc"
server_key_iv: "aa473de4bd15e13b70717757797040bd"
server_key_version: "encryptionkey@0"
I20260929 02:50:59.912767 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "ef079377d96e443e8f785d975f7074f5"
format_stamp: "Formatted at 2026-09-29 02:50:59 on dist-test-slave-6tjh"
server_key: "a867650a370742dc2ed13cb2a51b0bdc"
server_key_iv: "aa473de4bd15e13b70717757797040bd"
server_key_version: "encryptionkey@0"
I20260929 02:50:59.918459 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:50:59.922912 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:59.924880 18828 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:50:59.925071 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "ef079377d96e443e8f785d975f7074f5"
format_stamp: "Formatted at 2026-09-29 02:50:59 on dist-test-slave-6tjh"
server_key: "a867650a370742dc2ed13cb2a51b0bdc"
server_key_iv: "aa473de4bd15e13b70717757797040bd"
server_key_version: "encryptionkey@0"
I20260929 02:50:59.925284 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:50:59.944262 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:50:59.949421 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:50:59.949795 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:50:59.950887 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:50:59.952390 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:50:59.952471 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:59.952679 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:50:59.952747 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:50:59.989560 18828 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43323
I20260929 02:50:59.989624 18955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43323 every 8 connection(s)
I20260929 02:50:59.991014 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:50:59.997725 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18828
I20260929 02:50:59.997991 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:50:59.998469 388 external_mini_cluster.cc:1428] Setting key 824d4f201d2d68f604fb16988f3121f6
Sep 29 02:51:00 dist-test-slave-6tjh krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:00.014160 18958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:59.993911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48231 (local address 127.0.97.62:38833)
0929 02:50:59.994382 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 02:50:59.994386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:50:59.995456 (+ 1070us) server_negotiation.cc:407] Connection header received
0929 02:50:59.997437 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:50:59.997445 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:50:59.997556 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:50:59.997706 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:50:59.999432 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.000399 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.002475 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.003253 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.003414 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.007160 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:00.007169 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:00.007178 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:00.007226 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:00.010364 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.011510 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.011514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.011517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.011649 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.012212 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.012215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.012221 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.012515 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:00.012721 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.013714 (+ 993us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.013893 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":243,"threads_started":1}
I20260929 02:51:00.015934 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:50:59.994543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38833 (local address 127.0.97.2:48231)
0929 02:50:59.995285 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 02:50:59.995329 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 02:50:59.997122 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:50:59.997984 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:50:59.997993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:50:59.998657 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:50:59.999273 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:50:59.999291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.000633 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.000637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.002266 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.002282 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.003509 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.003516 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.003747 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.004490 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:00.004518 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:00.006878 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:00.010534 (+ 3656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:00.010548 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:00.010572 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:00.011322 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:00.011773 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:00.011781 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:00.011789 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:00.012075 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:00.012717 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:00.012731 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:00.013278 (+ 547us) client_negotiation.cc:769] Sending connection context
0929 02:51:00.013624 (+ 346us) client_negotiation.cc:240] Negotiation successful
0929 02:51:00.014291 (+ 667us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":278,"threads_started":1}
I20260929 02:51:00.017596 18956 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38833
I20260929 02:51:00.017918 18956 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:00.018673 18956 heartbeater.cc:511] Master 127.0.97.62:38833 requested a full tablet report, sending...
I20260929 02:51:00.020195 18633 ts_manager.cc:194] Registered new tserver with Master: ef079377d96e443e8f785d975f7074f5 (127.0.97.2:43323)
I20260929 02:51:00.021101 18633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48231
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:51:00.061434 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38833
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:00.406337 18962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:00.406646 18962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:00.406821 18962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:00.414268 18962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:00.414386 18962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:00.414592 18962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:51:00.424949 18962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:38833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:00.426379 18962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:00.427733 18962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:00.442476 18970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:00.442860 18968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:00.442883 18967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:00.443553 18962 server_base.cc:1034] running on GCE node
I20260929 02:51:00.444180 18962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:00.445216 18962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:00.446447 18962 hybrid_clock.cc:648] HybridClock initialized: now 1790650260446391 us; error 34 us; skew 500 ppm
I20260929 02:51:00.446816 18962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:00 dist-test-slave-6tjh krb5kdc[18588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:00.450244 18962 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:51:00.451831 18962 webserver.cc:460] Webserver started at http://127.0.97.3:38021/ using document root <none> and password file <none>
I20260929 02:51:00.452620 18962 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:00.452780 18962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:00.453102 18962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:00.456334 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "1b6767ffe63342edac5dd9975896b08b"
format_stamp: "Formatted at 2026-09-29 02:51:00 on dist-test-slave-6tjh"
server_key: "2ccfa2e337965688c3b80a88a41eb05f"
server_key_iv: "62dbb485bfa26655cc03685987d3a0f5"
server_key_version: "encryptionkey@0"
I20260929 02:51:00.457281 18962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "1b6767ffe63342edac5dd9975896b08b"
format_stamp: "Formatted at 2026-09-29 02:51:00 on dist-test-slave-6tjh"
server_key: "2ccfa2e337965688c3b80a88a41eb05f"
server_key_iv: "62dbb485bfa26655cc03685987d3a0f5"
server_key_version: "encryptionkey@0"
I20260929 02:51:00.463111 18962 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:51:00.467525 18977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:00.469280 18962 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 02:51:00.469506 18962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "1b6767ffe63342edac5dd9975896b08b"
format_stamp: "Formatted at 2026-09-29 02:51:00 on dist-test-slave-6tjh"
server_key: "2ccfa2e337965688c3b80a88a41eb05f"
server_key_iv: "62dbb485bfa26655cc03685987d3a0f5"
server_key_version: "encryptionkey@0"
I20260929 02:51:00.469762 18962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:00.485049 18822 heartbeater.cc:503] Master 127.0.97.62:38833 was elected leader, sending a full tablet report...
I20260929 02:51:00.487305 18962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:00.491945 18962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:00.492306 18962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:00.493459 18962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:00.494834 18962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:00.494908 18962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:00.495003 18962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:00.495047 18962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:00.530180 18962 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42597
I20260929 02:51:00.530233 19089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42597 every 8 connection(s)
I20260929 02:51:00.531574 18962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:51:00.537823 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 18962
I20260929 02:51:00.538022 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:51:00.538589 388 external_mini_cluster.cc:1428] Setting key 06e588c91dbc7ca2e99220a28e349a75
Sep 29 02:51:00 dist-test-slave-6tjh krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:00.555145 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.534287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42141 (local address 127.0.97.62:38833)
0929 02:51:00.534844 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.534852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.535874 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 02:51:00.538350 (+ 2476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.538355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.538434 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.538542 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:00.540743 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.541914 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.543541 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.543943 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.544039 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.547786 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:00.547791 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:00.547795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:00.547831 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:00.550841 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.551697 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.551702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.551704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.551838 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.552648 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.552654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.552657 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.553066 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:00.553276 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.554435 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.554733 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":277,"threads_started":1}
I20260929 02:51:00.556231 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.534833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38833 (local address 127.0.97.3:42141)
0929 02:51:00.535630 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:00.535676 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:00.537987 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:00.538861 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:00.538874 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:00.539642 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:00.540521 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.540548 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.542097 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.542101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.543407 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.543420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.544152 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.544157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.544360 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.545096 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:00.545119 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:00.547541 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:00.551023 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:00.551040 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:00.551055 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:00.551534 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:00.552074 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:00.552081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:00.552085 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:00.552406 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:00.553235 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:00.553242 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:00.553862 (+ 620us) client_negotiation.cc:769] Sending connection context
0929 02:51:00.554238 (+ 376us) client_negotiation.cc:240] Negotiation successful
0929 02:51:00.554892 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":311,"threads_started":1}
I20260929 02:51:00.557829 19090 heartbeater.cc:347] Connected to a master server at 127.0.97.62:38833
I20260929 02:51:00.558203 19090 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:00.558959 19090 heartbeater.cc:511] Master 127.0.97.62:38833 requested a full tablet report, sending...
I20260929 02:51:00.560477 18633 ts_manager.cc:194] Registered new tserver with Master: 1b6767ffe63342edac5dd9975896b08b (127.0.97.3:42597)
I20260929 02:51:00.561388 18633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42141
I20260929 02:51:00.568163 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:51:00.586969 19092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.572622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.0.97.62:38833)
0929 02:51:00.572799 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.572803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.573393 (+ 590us) server_negotiation.cc:407] Connection header received
0929 02:51:00.573632 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.573636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.573704 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.573813 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:00.575263 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.576006 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.577391 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.577815 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.577952 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.580854 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:00.580859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:00.580863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:00.580899 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:00.583799 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.584794 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.584799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.584801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.584930 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.585434 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.585438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.585440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.585762 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:00.586041 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.586448 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.586678 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:51:00.591138 18633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:51:00.598205 18633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:51:00.626019 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.611983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.0.97.1:39279)
0929 02:51:00.613497 (+ 1514us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.613508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.613541 (+ 33us) server_negotiation.cc:407] Connection header received
0929 02:51:00.613646 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.613655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.613932 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.614136 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:00.617413 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.619518 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.622798 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.624427 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.624608 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.624875 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.625468 (+ 593us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.625668 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1331,"thread_start_us":1165,"threads_started":1}
I20260929 02:51:00.626011 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.613897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.0.97.2:43323)
0929 02:51:00.617442 (+ 3545us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.617468 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.617502 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:51:00.617601 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.617609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.617860 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.618086 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:00.618780 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.620324 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.623339 (+ 3015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.624903 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.625065 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.625321 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.625525 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.625693 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3325,"thread_start_us":3197,"threads_started":1}
I20260929 02:51:00.626067 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.613449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43323 (local address 127.0.0.1:36126)
0929 02:51:00.616646 (+ 3197us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:00.616670 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:00.616836 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:00.618070 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:00.618074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:00.618101 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:00.618557 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.618576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.620482 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.620490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.623094 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.623113 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.625125 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.625134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.625322 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.625361 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:51:00.625481 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 02:51:00.625617 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3020,"mutex_wait_us":75,"thread_start_us":2821,"threads_started":1}
I20260929 02:51:00.626619 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.611525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42597 (local address 127.0.0.1:54532)
0929 02:51:00.616233 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:00.616257 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:00.616380 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:00.618429 (+ 2049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:00.618434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:00.618468 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:00.618925 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.618937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.620597 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.620603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.623370 (+ 2767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.623385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.625691 (+ 2306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.625698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.625882 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.625910 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 02:51:00.626024 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 02:51:00.626165 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4543,"mutex_wait_us":241,"thread_start_us":4388,"threads_started":1}
I20260929 02:51:00.626643 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.611401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39279 (local address 127.0.0.1:48048)
0929 02:51:00.612194 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:00.612243 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:00.612423 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:00.616647 (+ 4224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:00.616658 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:00.616709 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:00.617180 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.617193 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.619698 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.619705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.622569 (+ 2864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.622592 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.624671 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.624678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.624866 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.624922 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 02:51:00.625060 (+ 138us) client_negotiation.cc:240] Negotiation successful
0929 02:51:00.625307 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":19,"thread_start_us":1245,"threads_started":1}
I20260929 02:51:00.627434 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.612488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.0.97.3:42597)
0929 02:51:00.617822 (+ 5334us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.617834 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.617869 (+ 35us) server_negotiation.cc:407] Connection header received
0929 02:51:00.617971 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.617979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.618220 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.618406 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:00.619135 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.620383 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.623571 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.625124 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.625332 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.625564 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.626994 (+ 1430us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.627185 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5157,"thread_start_us":4965,"threads_started":1}
I20260929 02:51:00.630707 18757 tablet_service.cc:1467] Processing CreateTablet for tablet f5cbe8ec60cb4a4f8c17197773c036a4 (DEFAULT_TABLE table=test-table [id=bb745dacd248454e847ef9522aeb0388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:00.630769 18891 tablet_service.cc:1467] Processing CreateTablet for tablet f5cbe8ec60cb4a4f8c17197773c036a4 (DEFAULT_TABLE table=test-table [id=bb745dacd248454e847ef9522aeb0388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:00.631399 19025 tablet_service.cc:1467] Processing CreateTablet for tablet f5cbe8ec60cb4a4f8c17197773c036a4 (DEFAULT_TABLE table=test-table [id=bb745dacd248454e847ef9522aeb0388]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:00.632149 18757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cbe8ec60cb4a4f8c17197773c036a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:00.632213 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cbe8ec60cb4a4f8c17197773c036a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:00.632766 19025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5cbe8ec60cb4a4f8c17197773c036a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:00.645649 19109 tablet_bootstrap.cc:492] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Bootstrap starting.
I20260929 02:51:00.647833 19111 tablet_bootstrap.cc:492] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Bootstrap starting.
I20260929 02:51:00.649642 19109 tablet_bootstrap.cc:654] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:00.651119 19109 log.cc:824] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:00.651777 19111 tablet_bootstrap.cc:654] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:00.653353 19111 log.cc:824] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:00.654345 19110 tablet_bootstrap.cc:492] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Bootstrap starting.
I20260929 02:51:00.654951 19109 tablet_bootstrap.cc:492] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: No bootstrap required, opened a new log
I20260929 02:51:00.655262 19109 ts_tablet_manager.cc:1397] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 02:51:00.659339 19110 tablet_bootstrap.cc:654] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:00.660668 19110 log.cc:824] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:00.662541 19111 tablet_bootstrap.cc:492] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: No bootstrap required, opened a new log
I20260929 02:51:00.662842 19111 ts_tablet_manager.cc:1397] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260929 02:51:00.663831 19110 tablet_bootstrap.cc:492] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: No bootstrap required, opened a new log
I20260929 02:51:00.663717 19109 raft_consensus.cc:348] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.664137 19110 ts_tablet_manager.cc:1397] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260929 02:51:00.664206 19109 raft_consensus.cc:374] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:00.664341 19109 raft_consensus.cc:729] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bb069f608b94d4da0871b507a602fb6, State: Initialized, Role: FOLLOWER
I20260929 02:51:00.665165 19109 consensus_queue.cc:260] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.666983 19109 ts_tablet_manager.cc:1428] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260929 02:51:00.672473 19111 raft_consensus.cc:348] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.673010 19111 raft_consensus.cc:374] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:00.673130 19111 raft_consensus.cc:729] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6767ffe63342edac5dd9975896b08b, State: Initialized, Role: FOLLOWER
I20260929 02:51:00.673941 19111 consensus_queue.cc:260] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.675072 19090 heartbeater.cc:503] Master 127.0.97.62:38833 was elected leader, sending a full tablet report...
I20260929 02:51:00.674953 19110 raft_consensus.cc:348] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.675384 19110 raft_consensus.cc:374] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:00.675480 19110 raft_consensus.cc:729] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef079377d96e443e8f785d975f7074f5, State: Initialized, Role: FOLLOWER
I20260929 02:51:00.676107 19110 consensus_queue.cc:260] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.677188 19111 ts_tablet_manager.cc:1428] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 02:51:00.677349 18956 heartbeater.cc:503] Master 127.0.97.62:38833 was elected leader, sending a full tablet report...
I20260929 02:51:00.678357 19110 ts_tablet_manager.cc:1428] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260929 02:51:00.690115 18823 tablet.cc:2369] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:51:00.743211 18957 tablet.cc:2369] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:51:00.783324 19091 tablet.cc:2369] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:00.806706 19116 raft_consensus.cc:482] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:51:00.807113 19116 raft_consensus.cc:504] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.808977 19116 leader_election.cc:288] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bb069f608b94d4da0871b507a602fb6 (127.0.97.1:39279), ef079377d96e443e8f785d975f7074f5 (127.0.97.2:43323)
I20260929 02:51:00.814760 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.809567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58895 (local address 127.0.97.1:39279)
0929 02:51:00.809808 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.809814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.809835 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:51:00.810030 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.810038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.810147 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.810263 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:00.810755 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.811478 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.813126 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.813885 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.814036 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.814150 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.814390 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.814516 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:51:00.814785 19093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.809392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39279 (local address 127.0.97.3:58895)
0929 02:51:00.809656 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:00.809693 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:00.809858 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:00.810435 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:00.810438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:00.810458 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:00.810609 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.810623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.811632 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.811644 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.812962 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.812977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.814041 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.814047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.814235 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.814259 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 02:51:00.814363 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 02:51:00.814494 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:51:00.815557 19118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.809798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:43323 (local address 127.0.97.3:40097)
0929 02:51:00.810540 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:00.810554 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:00.810692 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:00.811020 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:00.811024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:00.811048 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:00.811355 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.811363 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.812347 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.812352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.813702 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:00.813713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.814852 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.814858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:00.815048 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.815077 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 02:51:00.815165 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 02:51:00.815271 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":427,"threads_started":1}
I20260929 02:51:00.815579 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.810128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40097 (local address 127.0.97.2:43323)
0929 02:51:00.810349 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.810355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.810655 (+ 300us) server_negotiation.cc:407] Connection header received
0929 02:51:00.810840 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.810844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.810902 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.811005 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:00.811519 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.812195 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.813866 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.814636 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.814787 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.814883 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.815221 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.815355 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:51:00.816054 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cbe8ec60cb4a4f8c17197773c036a4" candidate_uuid: "1b6767ffe63342edac5dd9975896b08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb069f608b94d4da0871b507a602fb6" is_pre_election: true
I20260929 02:51:00.816699 18777 raft_consensus.cc:2454] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b6767ffe63342edac5dd9975896b08b in term 0.
I20260929 02:51:00.816694 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cbe8ec60cb4a4f8c17197773c036a4" candidate_uuid: "1b6767ffe63342edac5dd9975896b08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef079377d96e443e8f785d975f7074f5" is_pre_election: true
I20260929 02:51:00.817224 18911 raft_consensus.cc:2454] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b6767ffe63342edac5dd9975896b08b in term 0.
I20260929 02:51:00.817395 18979 leader_election.cc:302] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6767ffe63342edac5dd9975896b08b, 2bb069f608b94d4da0871b507a602fb6; no voters:
I20260929 02:51:00.817813 19116 raft_consensus.cc:2781] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:51:00.817962 19116 raft_consensus.cc:482] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:51:00.818081 19116 raft_consensus.cc:3037] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:00.820343 19116 raft_consensus.cc:504] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.821089 19116 leader_election.cc:288] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [CANDIDATE]: Term 1 election: Requested vote from peers 2bb069f608b94d4da0871b507a602fb6 (127.0.97.1:39279), ef079377d96e443e8f785d975f7074f5 (127.0.97.2:43323)
I20260929 02:51:00.821493 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cbe8ec60cb4a4f8c17197773c036a4" candidate_uuid: "1b6767ffe63342edac5dd9975896b08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bb069f608b94d4da0871b507a602fb6"
I20260929 02:51:00.821529 18911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5cbe8ec60cb4a4f8c17197773c036a4" candidate_uuid: "1b6767ffe63342edac5dd9975896b08b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef079377d96e443e8f785d975f7074f5"
I20260929 02:51:00.821691 18777 raft_consensus.cc:3037] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:00.821811 18911 raft_consensus.cc:3037] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:00.824020 18777 raft_consensus.cc:2454] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6767ffe63342edac5dd9975896b08b in term 1.
I20260929 02:51:00.824083 18911 raft_consensus.cc:2454] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6767ffe63342edac5dd9975896b08b in term 1.
I20260929 02:51:00.824404 18979 leader_election.cc:302] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6767ffe63342edac5dd9975896b08b, 2bb069f608b94d4da0871b507a602fb6; no voters:
I20260929 02:51:00.824740 19116 raft_consensus.cc:2781] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:00.825191 19116 raft_consensus.cc:686] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 1 LEADER]: Becoming Leader. State: Replica: 1b6767ffe63342edac5dd9975896b08b, State: Running, Role: LEADER
I20260929 02:51:00.825734 19116 consensus_queue.cc:237] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } }
I20260929 02:51:00.830897 18631 catalog_manager.cc:5495] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b6767ffe63342edac5dd9975896b08b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "1b6767ffe63342edac5dd9975896b08b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b6767ffe63342edac5dd9975896b08b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 } health_report { overall_health: UNKNOWN } } }
I20260929 02:51:00.867288 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.861570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.0.97.3:42597)
0929 02:51:00.861801 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.861807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.861827 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:51:00.862090 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.862094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.862176 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.862341 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:00.862750 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.863493 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.864966 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.865389 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.865533 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.866154 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:00.866675 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:00.866870 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.866986 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.867043 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:51:00.876796 18777 raft_consensus.cc:1261] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 1 FOLLOWER]: Refusing update from remote peer 1b6767ffe63342edac5dd9975896b08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:00.876914 18911 raft_consensus.cc:1261] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 1 FOLLOWER]: Refusing update from remote peer 1b6767ffe63342edac5dd9975896b08b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:00.877859 19116 consensus_queue.cc:1035] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bb069f608b94d4da0871b507a602fb6" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:00.878324 19119 consensus_queue.cc:1035] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef079377d96e443e8f785d975f7074f5" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 43323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:00.893153 19122 mvcc.cc:205] Tried to move back new op lower bound from 7334503468539699200 to 7334503468345942016. Current Snapshot: MvccSnapshot[applied={T|T < 7334503468539699200}]
I20260929 02:51:00.906711 18631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:00.907066 18631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:00.911371 18631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b1abcb477a64fc1a19f1dfbdba6fc26: Sending DeleteTablet for 3 replicas of tablet f5cbe8ec60cb4a4f8c17197773c036a4
I20260929 02:51:00.912762 19025 tablet_service.cc:1514] Processing DeleteTablet for tablet f5cbe8ec60cb4a4f8c17197773c036a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54532
I20260929 02:51:00.912897 18757 tablet_service.cc:1514] Processing DeleteTablet for tablet f5cbe8ec60cb4a4f8c17197773c036a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48048
I20260929 02:51:00.913216 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet f5cbe8ec60cb4a4f8c17197773c036a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36126
I20260929 02:51:00.915655 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18694
I20260929 02:51:00.921404 19133 tablet_replica.cc:330] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: stopping tablet replica
I20260929 02:51:00.922293 19133 raft_consensus.cc:2229] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:00.923229 19133 raft_consensus.cc:2258] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:00.925525 19131 tablet_replica.cc:330] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: stopping tablet replica
I20260929 02:51:00.926226 19132 tablet_replica.cc:330] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: stopping tablet replica
I20260929 02:51:00.926882 19132 raft_consensus.cc:2229] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:00.927508 19131 raft_consensus.cc:2229] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:51:00.928243 19132 raft_consensus.cc:2258] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:00.928321 19133 ts_tablet_manager.cc:1905] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:00.929132 19131 raft_consensus.cc:2258] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:00.931605 19132 ts_tablet_manager.cc:1905] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:00.934657 19131 ts_tablet_manager.cc:1905] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:00.937554 19133 ts_tablet_manager.cc:1918] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:00.937853 19133 log.cc:1196] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/f5cbe8ec60cb4a4f8c17197773c036a4
I20260929 02:51:00.938527 19133 ts_tablet_manager.cc:1939] T f5cbe8ec60cb4a4f8c17197773c036a4 P ef079377d96e443e8f785d975f7074f5: Deleting consensus metadata
Sep 29 02:51:00 dist-test-slave-6tjh krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:51:00.940757 18618 catalog_manager.cc:4841] TS ef079377d96e443e8f785d975f7074f5 (127.0.97.2:43323): tablet f5cbe8ec60cb4a4f8c17197773c036a4 (table test-table [id=bb745dacd248454e847ef9522aeb0388]) successfully deleted
I20260929 02:51:00.943581 19131 ts_tablet_manager.cc:1918] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:00.943881 19131 log.cc:1196] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/f5cbe8ec60cb4a4f8c17197773c036a4
I20260929 02:51:00.945004 19131 ts_tablet_manager.cc:1939] T f5cbe8ec60cb4a4f8c17197773c036a4 P 1b6767ffe63342edac5dd9975896b08b: Deleting consensus metadata
I20260929 02:51:00.946170 19132 ts_tablet_manager.cc:1918] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:00.946411 19132 log.cc:1196] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/f5cbe8ec60cb4a4f8c17197773c036a4
I20260929 02:51:00.947038 19132 ts_tablet_manager.cc:1939] T f5cbe8ec60cb4a4f8c17197773c036a4 P 2bb069f608b94d4da0871b507a602fb6: Deleting consensus metadata
I20260929 02:51:00.947717 18617 catalog_manager.cc:4841] TS 1b6767ffe63342edac5dd9975896b08b (127.0.97.3:42597): tablet f5cbe8ec60cb4a4f8c17197773c036a4 (table test-table [id=bb745dacd248454e847ef9522aeb0388]) successfully deleted
I20260929 02:51:00.949120 18618 catalog_manager.cc:4841] TS 2bb069f608b94d4da0871b507a602fb6 (127.0.97.1:39279): tablet f5cbe8ec60cb4a4f8c17197773c036a4 (table test-table [id=bb745dacd248454e847ef9522aeb0388]) successfully deleted
I20260929 02:51:00.950065 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:00.917454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.0.97.1:39279)
0929 02:51:00.917720 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:00.917729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:00.925754 (+ 8025us) server_negotiation.cc:407] Connection header received
0929 02:51:00.926101 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:00.926110 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:00.926212 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:00.926300 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:00.928271 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.929400 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.931967 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:00.932697 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:00.932807 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:00.942056 (+ 9249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:00.942078 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:00.942099 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:00.942160 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:00.946429 (+ 4269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.947492 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.947503 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.947506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.947701 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:00.948203 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:00.948207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:00.948212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:00.948756 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:00.949055 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:00.949452 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 02:51:00.949700 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":97}
I20260929 02:51:00.950784 18817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:01.043651 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18694
I20260929 02:51:01.062088 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18828
Sep 29 02:51:01 dist-test-slave-6tjh krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:51:01.082398 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:01.062983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36140 (local address 127.0.97.2:43323)
0929 02:51:01.063141 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:01.063147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:01.063181 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:51:01.063376 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:01.063381 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:01.063469 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:01.063552 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:01.065493 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:01.066295 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:01.068389 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:01.068994 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:01.069112 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:01.075055 (+ 5943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:01.075071 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:01.075087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:01.075131 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:01.078977 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:01.079759 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:01.079771 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:01.079777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:01.079957 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:01.080358 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:01.080364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:01.080368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:01.080955 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:01.081287 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:01.081695 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 02:51:01.081969 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 02:51:01.083184 18951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:01.206815 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18828
I20260929 02:51:01.232066 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18962
Sep 29 02:51:01 dist-test-slave-6tjh krb5kdc[18588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:51:01.250851 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:01.233053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54552 (local address 127.0.97.3:42597)
0929 02:51:01.233311 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:01.233321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:01.233341 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:51:01.233492 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:01.233497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:01.233572 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:01.233688 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:01.235046 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:01.235966 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:01.237812 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:01.238308 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:01.238465 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:01.243601 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:01.243615 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:01.243628 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:01.243671 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:01.247102 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:01.248156 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:01.248163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:01.248165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:01.248339 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:01.249092 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:01.249096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:01.249098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:01.249482 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:01.249733 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:01.250199 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 02:51:01.250401 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260929 02:51:01.251659 19085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:01.378971 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18962
I20260929 02:51:01.398679 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 18601
I20260929 02:51:01.399839 18663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:01.514014 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 18601
2026-09-29T02:51:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4634 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T02:51:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:01Z Disabled control of system clock
I20260929 02:51:01.594483 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33797
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:51:01.948046 19142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:01.948370 19142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:01.948494 19142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:01.956094 19142 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:01.956224 19142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:01.956314 19142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:01.956408 19142 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:01.956490 19142 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:01.966285 19142 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:01.967661 19142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:01.969640 19142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:01.982101 19147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:01.982216 19148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:01.982144 19150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:01.986119 19142 server_base.cc:1034] running on GCE node
I20260929 02:51:01.986981 19142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:01.988493 19142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:01.989763 19142 hybrid_clock.cc:648] HybridClock initialized: now 1790650261989714 us; error 54 us; skew 500 ppm
I20260929 02:51:01.990129 19142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:01.992709 19142 webserver.cc:460] Webserver started at http://127.0.97.62:34481/ using document root <none> and password file <none>
I20260929 02:51:01.993426 19142 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:01.993570 19142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:01.993893 19142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:01.996951 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "fc1e487d451843bb87dae58fdf90f911"
format_stamp: "Formatted at 2026-09-29 02:51:01 on dist-test-slave-6tjh"
server_key: "e9718d701f1cca0b407c817c7cb772cd"
server_key_iv: "7488f0808c4d7d059f93c52b43e23adb"
server_key_version: "encryptionkey@0"
I20260929 02:51:01.997778 19142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "fc1e487d451843bb87dae58fdf90f911"
format_stamp: "Formatted at 2026-09-29 02:51:01 on dist-test-slave-6tjh"
server_key: "e9718d701f1cca0b407c817c7cb772cd"
server_key_iv: "7488f0808c4d7d059f93c52b43e23adb"
server_key_version: "encryptionkey@0"
I20260929 02:51:02.003248 19142 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:51:02.007064 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:02.008754 19142 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:51:02.009112 19142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "fc1e487d451843bb87dae58fdf90f911"
format_stamp: "Formatted at 2026-09-29 02:51:01 on dist-test-slave-6tjh"
server_key: "e9718d701f1cca0b407c817c7cb772cd"
server_key_iv: "7488f0808c4d7d059f93c52b43e23adb"
server_key_version: "encryptionkey@0"
I20260929 02:51:02.009406 19142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:02.028656 19142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:02.029599 19142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:02.029907 19142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:02.049356 19142 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33797
I20260929 02:51:02.049378 19207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33797 every 8 connection(s)
I20260929 02:51:02.050806 19142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:02.054628 19208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:02.061080 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 19142
I20260929 02:51:02.061269 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:02.061798 388 external_mini_cluster.cc:1428] Setting key c35ba75a3536e0216a56ab56569d58e7
I20260929 02:51:02.066535 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911: Bootstrap starting.
I20260929 02:51:02.069927 19208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:02.071012 19208 log.cc:824] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:02.074096 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911: No bootstrap required, opened a new log
I20260929 02:51:02.075963 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:02.063876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33528 (local address 127.0.97.62:33797)
0929 02:51:02.065011 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:02.065031 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:02.065071 (+ 40us) server_negotiation.cc:407] Connection header received
0929 02:51:02.066899 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:02.066923 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:02.067409 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:02.067817 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:02.068427 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:02.069867 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:02.071335 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:02.072118 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:02.072327 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:02.073583 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:02.073590 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:02.073605 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:02.074144 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:02.074252 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:02.074281 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:02.074414 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:02.074641 (+ 227us) server_negotiation.cc:299] Negotiation successful
0929 02:51:02.074958 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":384,"threads_started":1}
I20260929 02:51:02.082320 19208 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1e487d451843bb87dae58fdf90f911" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:51:02.082846 19208 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:02.082980 19208 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc1e487d451843bb87dae58fdf90f911, State: Initialized, Role: FOLLOWER
I20260929 02:51:02.083647 19208 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1e487d451843bb87dae58fdf90f911" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:51:02.083887 19208 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:02.084038 19208 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:02.084237 19208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:02.086534 19208 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1e487d451843bb87dae58fdf90f911" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:51:02.087077 19208 leader_election.cc:302] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc1e487d451843bb87dae58fdf90f911; no voters:
I20260929 02:51:02.087692 19208 leader_election.cc:288] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:02.087877 19213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:02.088629 19213 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [term 1 LEADER]: Becoming Leader. State: Replica: fc1e487d451843bb87dae58fdf90f911, State: Running, Role: LEADER
I20260929 02:51:02.089206 19213 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1e487d451843bb87dae58fdf90f911" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } }
I20260929 02:51:02.089931 19208 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:02.092492 19214 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc1e487d451843bb87dae58fdf90f911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1e487d451843bb87dae58fdf90f911" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } } }
I20260929 02:51:02.092537 19215 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc1e487d451843bb87dae58fdf90f911. Latest consensus state: current_term: 1 leader_uuid: "fc1e487d451843bb87dae58fdf90f911" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc1e487d451843bb87dae58fdf90f911" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33797 } } }
I20260929 02:51:02.092980 19214 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:02.093039 19215 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:02.095542 19220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:02.100955 19220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:02.111104 19220 catalog_manager.cc:1347] Generated new cluster ID: 4051f426a2fe4bcfa73e715fdfbfc1be
I20260929 02:51:02.111243 19220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:02.119740 19220 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:02.121183 19220 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:02.132242 19220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911: Generated new TSK 0
I20260929 02:51:02.133241 19220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:02.141938 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33797
--builtin_ntp_servers=127.0.97.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:51:02.516718 19232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:02.517041 19232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:02.517201 19232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:02.525234 19232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:02.525353 19232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:02.525525 19232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:51:02.535636 19232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:02.537034 19232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:02.538868 19232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:02.553133 19237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:02.553094 19238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:02.553357 19240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:02.554914 19232 server_base.cc:1034] running on GCE node
I20260929 02:51:02.555721 19232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:02.556867 19232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:02.558131 19232 hybrid_clock.cc:648] HybridClock initialized: now 1790650262558039 us; error 64 us; skew 500 ppm
I20260929 02:51:02.558483 19232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:02.561385 19232 webserver.cc:460] Webserver started at http://127.0.97.1:41319/ using document root <none> and password file <none>
I20260929 02:51:02.562175 19232 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:02.562316 19232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:02.562664 19232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:02.565837 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "33a5db07002c4354bf3e5c92d4b9e4c5"
format_stamp: "Formatted at 2026-09-29 02:51:02 on dist-test-slave-6tjh"
server_key: "35f90c86456987540a505e4efa161ffc"
server_key_iv: "543506e9db3e6b9dbefeffe7f1ab9fc4"
server_key_version: "encryptionkey@0"
I20260929 02:51:02.567000 19232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "33a5db07002c4354bf3e5c92d4b9e4c5"
format_stamp: "Formatted at 2026-09-29 02:51:02 on dist-test-slave-6tjh"
server_key: "35f90c86456987540a505e4efa161ffc"
server_key_iv: "543506e9db3e6b9dbefeffe7f1ab9fc4"
server_key_version: "encryptionkey@0"
I20260929 02:51:02.574012 19232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260929 02:51:02.578022 19246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:02.579501 19232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:51:02.579704 19232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "33a5db07002c4354bf3e5c92d4b9e4c5"
format_stamp: "Formatted at 2026-09-29 02:51:02 on dist-test-slave-6tjh"
server_key: "35f90c86456987540a505e4efa161ffc"
server_key_iv: "543506e9db3e6b9dbefeffe7f1ab9fc4"
server_key_version: "encryptionkey@0"
I20260929 02:51:02.580004 19232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:02.595245 19232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:02.596249 19232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:02.596616 19232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:02.597643 19232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:02.599207 19232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:02.599329 19232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:02.599472 19232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:02.599563 19232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:02.634049 19232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:33647
I20260929 02:51:02.634099 19358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:33647 every 8 connection(s)
I20260929 02:51:02.635406 19232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:51:02.642491 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 19232
I20260929 02:51:02.642679 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:51:02.643133 388 external_mini_cluster.cc:1428] Setting key 1fd326ac6f43ad7e207a7464d03c35d6
I20260929 02:51:02.646288 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33797
--builtin_ntp_servers=127.0.97.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:51:02.649526 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:02.637950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48857 (local address 127.0.97.62:33797)
0929 02:51:02.638409 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:02.638414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:02.639676 (+ 1262us) server_negotiation.cc:407] Connection header received
0929 02:51:02.641600 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:02.641606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:02.641707 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:02.641832 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:02.643130 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:02.644096 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:02.646351 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:02.646861 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:02.646975 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:02.648692 (+ 1717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:02.648696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:02.648700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:02.648718 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:02.648758 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:02.648769 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:02.648842 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:02.649153 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 02:51:02.649241 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":238,"threads_started":1}
I20260929 02:51:02.651062 19362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:02.638582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33797 (local address 127.0.97.1:48857)
0929 02:51:02.639496 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:02.639537 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:02.641314 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:02.642095 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:02.642104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:02.642205 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:02.642917 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:02.642945 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:02.644263 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:02.644267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:02.646193 (+ 1926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:02.646213 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:02.647104 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:02.647115 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:02.647475 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:02.648276 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:51:02.648281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:02.648341 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:51:02.648453 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:02.648871 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:02.648885 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:02.648895 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 02:51:02.649084 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 02:51:02.649548 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":254,"threads_started":1}
I20260929 02:51:02.652884 19359 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33797
I20260929 02:51:02.653270 19359 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:02.654083 19359 heartbeater.cc:511] Master 127.0.97.62:33797 requested a full tablet report, sending...
I20260929 02:51:02.656487 19173 ts_manager.cc:194] Registered new tserver with Master: 33a5db07002c4354bf3e5c92d4b9e4c5 (127.0.97.1:33647)
I20260929 02:51:02.658094 19173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:48857
W20260929 02:51:02.988379 19363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:02.988801 19363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:02.988976 19363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:02.996904 19363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:02.997054 19363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:02.997228 19363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:51:03.006660 19363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:03.008037 19363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:03.009883 19363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:03.023983 19369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:03.024147 19368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:03.024143 19371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:03.025197 19363 server_base.cc:1034] running on GCE node
I20260929 02:51:03.025861 19363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:03.026868 19363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:03.028111 19363 hybrid_clock.cc:648] HybridClock initialized: now 1790650263028051 us; error 54 us; skew 500 ppm
I20260929 02:51:03.028402 19363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:03.030819 19363 webserver.cc:460] Webserver started at http://127.0.97.2:44439/ using document root <none> and password file <none>
I20260929 02:51:03.031483 19363 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:03.031569 19363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:03.031829 19363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:03.034905 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "71f5a6b245534d608138c2096453d6ac"
format_stamp: "Formatted at 2026-09-29 02:51:03 on dist-test-slave-6tjh"
server_key: "428e905578aa5d1d553d92f31543dad7"
server_key_iv: "85811f212cb0b32a53922ea3eeabe8e8"
server_key_version: "encryptionkey@0"
I20260929 02:51:03.035647 19363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "71f5a6b245534d608138c2096453d6ac"
format_stamp: "Formatted at 2026-09-29 02:51:03 on dist-test-slave-6tjh"
server_key: "428e905578aa5d1d553d92f31543dad7"
server_key_iv: "85811f212cb0b32a53922ea3eeabe8e8"
server_key_version: "encryptionkey@0"
I20260929 02:51:03.040800 19363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:51:03.044452 19377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:03.046025 19363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:51:03.046218 19363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "71f5a6b245534d608138c2096453d6ac"
format_stamp: "Formatted at 2026-09-29 02:51:03 on dist-test-slave-6tjh"
server_key: "428e905578aa5d1d553d92f31543dad7"
server_key_iv: "85811f212cb0b32a53922ea3eeabe8e8"
server_key_version: "encryptionkey@0"
I20260929 02:51:03.046464 19363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:03.093075 19363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:03.093930 19363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:03.094221 19363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:03.095340 19363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:03.097078 19363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:03.097200 19363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:03.097347 19363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:03.097435 19363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:03.134794 19363 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34623
I20260929 02:51:03.134833 19489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34623 every 8 connection(s)
I20260929 02:51:03.136122 19363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:51:03.146093 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 19363
I20260929 02:51:03.146301 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:51:03.146826 388 external_mini_cluster.cc:1428] Setting key 68a4ba7f528077377f17b8d93f69f0fd
I20260929 02:51:03.149868 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.138862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42153 (local address 127.0.97.62:33797)
0929 02:51:03.139067 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.139072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.140478 (+ 1406us) server_negotiation.cc:407] Connection header received
0929 02:51:03.142364 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.142368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.142456 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.142588 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:03.143751 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.144532 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.146635 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.147241 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.147408 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.149057 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:03.149064 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:03.149071 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:03.149100 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:03.149145 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:03.149166 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:03.149266 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.149544 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.149628 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:51:03.150272 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33797
--builtin_ntp_servers=127.0.97.20:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:51:03.151788 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.139358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33797 (local address 127.0.97.2:42153)
0929 02:51:03.140286 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:03.140328 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:03.142016 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:03.142818 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:03.142828 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:03.142931 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:03.143564 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.143582 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.144740 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.144744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.146441 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.146460 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.147485 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.147490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.147850 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.148717 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:51:03.148720 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:03.148790 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:51:03.148873 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:03.149290 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:03.149303 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:03.149311 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 02:51:03.149531 (+ 220us) client_negotiation.cc:240] Negotiation successful
0929 02:51:03.150052 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":263,"threads_started":1}
I20260929 02:51:03.153872 19490 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33797
I20260929 02:51:03.154228 19490 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:03.155206 19490 heartbeater.cc:511] Master 127.0.97.62:33797 requested a full tablet report, sending...
I20260929 02:51:03.156870 19173 ts_manager.cc:194] Registered new tserver with Master: 71f5a6b245534d608138c2096453d6ac (127.0.97.2:34623)
I20260929 02:51:03.157696 19173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42153
W20260929 02:51:03.493505 19493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:03.493872 19493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:03.494032 19493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:03.501782 19493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:03.501940 19493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:03.502117 19493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:51:03.512137 19493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33797
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:03.513523 19493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:03.515626 19493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:03.529543 19501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:03.529600 19499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:03.529639 19498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:03.531775 19493 server_base.cc:1034] running on GCE node
I20260929 02:51:03.532419 19493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:03.533558 19493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:03.534760 19493 hybrid_clock.cc:648] HybridClock initialized: now 1790650263534691 us; error 67 us; skew 500 ppm
I20260929 02:51:03.535120 19493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:03.537837 19493 webserver.cc:460] Webserver started at http://127.0.97.3:38849/ using document root <none> and password file <none>
I20260929 02:51:03.538539 19493 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:03.538630 19493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:03.538892 19493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:03.542021 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "47bc14196891498eb8444d97b88414c4"
format_stamp: "Formatted at 2026-09-29 02:51:03 on dist-test-slave-6tjh"
server_key: "7fddc86fdad54fe05f70726d4361e101"
server_key_iv: "986a8c29864d88042fc62dea216071a8"
server_key_version: "encryptionkey@0"
I20260929 02:51:03.542858 19493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "47bc14196891498eb8444d97b88414c4"
format_stamp: "Formatted at 2026-09-29 02:51:03 on dist-test-slave-6tjh"
server_key: "7fddc86fdad54fe05f70726d4361e101"
server_key_iv: "986a8c29864d88042fc62dea216071a8"
server_key_version: "encryptionkey@0"
I20260929 02:51:03.548612 19493 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 02:51:03.552702 19507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:03.554298 19493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:51:03.554574 19493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "47bc14196891498eb8444d97b88414c4"
format_stamp: "Formatted at 2026-09-29 02:51:03 on dist-test-slave-6tjh"
server_key: "7fddc86fdad54fe05f70726d4361e101"
server_key_iv: "986a8c29864d88042fc62dea216071a8"
server_key_version: "encryptionkey@0"
I20260929 02:51:03.554850 19493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:03.591032 19493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:03.591988 19493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:03.592299 19493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:03.593472 19493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:03.595269 19493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:03.595399 19493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:03.595562 19493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:03.595659 19493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:03.631605 19493 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37957
I20260929 02:51:03.631654 19619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37957 every 8 connection(s)
I20260929 02:51:03.633001 19493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:51:03.639140 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 19493
I20260929 02:51:03.639338 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:51:03.639809 388 external_mini_cluster.cc:1428] Setting key 55f7e245f0ff65ca755a5847694bcb2b
I20260929 02:51:03.647372 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.635787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39355 (local address 127.0.97.62:33797)
0929 02:51:03.636033 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.636040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.637884 (+ 1844us) server_negotiation.cc:407] Connection header received
0929 02:51:03.640376 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.640382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.640481 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.640652 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:03.641961 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.643053 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.644717 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.645114 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.645213 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.646561 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:03.646566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:03.646570 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:03.646590 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:03.646625 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:03.646639 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:03.646725 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.646998 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.647089 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:51:03.648375 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.636441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33797 (local address 127.0.97.3:39355)
0929 02:51:03.637632 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:03.637685 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:03.640030 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:03.640898 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:03.640930 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:03.641072 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:03.641767 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.641795 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.643197 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.643201 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.644523 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.644535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.645313 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.645318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.645519 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.646218 (+ 699us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:51:03.646220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:03.646261 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:51:03.646340 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:03.646738 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:03.646746 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:03.646753 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 02:51:03.646913 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 02:51:03.647255 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":311,"threads_started":1}
I20260929 02:51:03.650087 19620 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33797
I20260929 02:51:03.650444 19620 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:03.651301 19620 heartbeater.cc:511] Master 127.0.97.62:33797 requested a full tablet report, sending...
I20260929 02:51:03.652874 19173 ts_manager.cc:194] Registered new tserver with Master: 47bc14196891498eb8444d97b88414c4 (127.0.97.3:37957)
I20260929 02:51:03.653815 19173 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39355
I20260929 02:51:03.655736 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:51:03.661422 19359 heartbeater.cc:503] Master 127.0.97.62:33797 was elected leader, sending a full tablet report...
I20260929 02:51:03.667114 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.660311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.0.97.62:33797)
0929 02:51:03.660581 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.660590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.660997 (+ 407us) server_negotiation.cc:407] Connection header received
0929 02:51:03.661103 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.661108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.661177 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.661271 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:03.661887 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.662863 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.664430 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.664950 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.665107 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.666335 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:03.666341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:03.666346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:03.666376 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:03.666408 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:03.666420 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:03.666567 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.666762 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.666823 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:51:03.671558 19172 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33536:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:51:03.678701 19172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:51:03.706317 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.693989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.0.97.3:37957)
0929 02:51:03.695246 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.695269 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.696303 (+ 1034us) server_negotiation.cc:407] Connection header received
0929 02:51:03.696618 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.696627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.696844 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.697009 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:03.698041 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.699553 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.703510 (+ 3957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.704870 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.704998 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.705278 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.705769 (+ 491us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.705934 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1020,"thread_start_us":828,"threads_started":1}
I20260929 02:51:03.706357 19632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.693377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37957 (local address 127.0.0.1:59582)
0929 02:51:03.696195 (+ 2818us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:03.696213 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:03.696399 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:03.697359 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:03.697366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:03.697392 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:03.697840 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.697856 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.699748 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.699753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.703335 (+ 3582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.703345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.705219 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.705224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.705441 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.705555 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 02:51:03.705715 (+ 160us) client_negotiation.cc:240] Negotiation successful
0929 02:51:03.705915 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2648,"mutex_wait_us":1337,"thread_start_us":422,"threads_started":1}
I20260929 02:51:03.706513 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.692987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40296 (local address 127.0.97.2:34623)
0929 02:51:03.695734 (+ 2747us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.695745 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.695783 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:51:03.696160 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.696170 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.696452 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.696682 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:03.697757 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.699370 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.703063 (+ 3693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.704798 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.704968 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.705251 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.705950 (+ 699us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.706154 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2571,"thread_start_us":2476,"threads_started":1}
I20260929 02:51:03.707513 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.692860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34623 (local address 127.0.0.1:40296)
0929 02:51:03.695578 (+ 2718us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:03.695630 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:03.695899 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:03.696959 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:03.696968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:03.696998 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:03.697504 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.697517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.699535 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.699540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.702841 (+ 3301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.702856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.705430 (+ 2574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.705438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.705697 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.705742 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 02:51:03.706944 (+ 1202us) client_negotiation.cc:240] Negotiation successful
0929 02:51:03.707102 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2347,"mutex_wait_us":774,"thread_start_us":2929,"threads_started":1}
I20260929 02:51:03.710513 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.696283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33647 (local address 127.0.0.1:35814)
0929 02:51:03.700609 (+ 4326us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:03.700640 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:03.700765 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:03.702032 (+ 1267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:03.702036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:03.702060 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:03.702511 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.702520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.704804 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.704810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.707374 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.707389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.709690 (+ 2301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.709698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.709889 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.709925 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 02:51:03.710045 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 02:51:03.710185 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4155,"thread_start_us":3989,"threads_started":1}
I20260929 02:51:03.710613 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.696728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35814 (local address 127.0.97.1:33647)
0929 02:51:03.701377 (+ 4649us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.701388 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.701419 (+ 31us) server_negotiation.cc:407] Connection header received
0929 02:51:03.701519 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.701527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.701782 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.701981 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:03.702809 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.704600 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.707967 (+ 3367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.709449 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.709589 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.709881 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.710099 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.710270 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4451,"thread_start_us":4296,"threads_started":1}
I20260929 02:51:03.710767 19555 tablet_service.cc:1467] Processing CreateTablet for tablet 997b7e359b224e51b840c4e58d9878df (DEFAULT_TABLE table=test-table [id=cd0fc62219cc479fa6223bae2b96c17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:03.711459 19425 tablet_service.cc:1467] Processing CreateTablet for tablet 997b7e359b224e51b840c4e58d9878df (DEFAULT_TABLE table=test-table [id=cd0fc62219cc479fa6223bae2b96c17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:03.711984 19555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997b7e359b224e51b840c4e58d9878df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:03.712870 19425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997b7e359b224e51b840c4e58d9878df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:03.716061 19294 tablet_service.cc:1467] Processing CreateTablet for tablet 997b7e359b224e51b840c4e58d9878df (DEFAULT_TABLE table=test-table [id=cd0fc62219cc479fa6223bae2b96c17f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:03.717568 19294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 997b7e359b224e51b840c4e58d9878df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:03.726519 19638 tablet_bootstrap.cc:492] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Bootstrap starting.
I20260929 02:51:03.730893 19638 tablet_bootstrap.cc:654] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:03.733165 19638 log.cc:824] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:03.733408 19639 tablet_bootstrap.cc:492] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Bootstrap starting.
I20260929 02:51:03.738083 19639 tablet_bootstrap.cc:654] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:03.739333 19639 log.cc:824] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:03.739725 19638 tablet_bootstrap.cc:492] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: No bootstrap required, opened a new log
I20260929 02:51:03.740033 19638 ts_tablet_manager.cc:1397] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260929 02:51:03.741310 19642 tablet_bootstrap.cc:492] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Bootstrap starting.
I20260929 02:51:03.745177 19639 tablet_bootstrap.cc:492] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: No bootstrap required, opened a new log
I20260929 02:51:03.745471 19639 ts_tablet_manager.cc:1397] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 02:51:03.746157 19642 tablet_bootstrap.cc:654] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:03.747655 19642 log.cc:824] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:03.751273 19642 tablet_bootstrap.cc:492] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: No bootstrap required, opened a new log
I20260929 02:51:03.751582 19642 ts_tablet_manager.cc:1397] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260929 02:51:03.751611 19638 raft_consensus.cc:348] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.752112 19638 raft_consensus.cc:374] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:03.752251 19638 raft_consensus.cc:729] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f5a6b245534d608138c2096453d6ac, State: Initialized, Role: FOLLOWER
I20260929 02:51:03.752998 19638 consensus_queue.cc:260] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.754274 19490 heartbeater.cc:503] Master 127.0.97.62:33797 was elected leader, sending a full tablet report...
I20260929 02:51:03.755125 19638 ts_tablet_manager.cc:1428] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260929 02:51:03.755322 19639 raft_consensus.cc:348] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.755839 19639 raft_consensus.cc:374] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:03.755959 19639 raft_consensus.cc:729] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bc14196891498eb8444d97b88414c4, State: Initialized, Role: FOLLOWER
I20260929 02:51:03.756816 19639 consensus_queue.cc:260] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.758116 19620 heartbeater.cc:503] Master 127.0.97.62:33797 was elected leader, sending a full tablet report...
I20260929 02:51:03.758967 19639 ts_tablet_manager.cc:1428] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
I20260929 02:51:03.764358 19642 raft_consensus.cc:348] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.764870 19642 raft_consensus.cc:374] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:03.765012 19642 raft_consensus.cc:729] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33a5db07002c4354bf3e5c92d4b9e4c5, State: Initialized, Role: FOLLOWER
I20260929 02:51:03.765633 19642 consensus_queue.cc:260] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.767263 19642 ts_tablet_manager.cc:1428] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260929 02:51:03.769657 19645 raft_consensus.cc:482] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:51:03.770017 19645 raft_consensus.cc:504] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.772208 19645 leader_election.cc:288] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71f5a6b245534d608138c2096453d6ac (127.0.97.2:34623), 33a5db07002c4354bf3e5c92d4b9e4c5 (127.0.97.1:33647)
I20260929 02:51:03.778489 19622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.772728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34623 (local address 127.0.97.3:41495)
0929 02:51:03.772969 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:03.772992 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:03.773138 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:03.773602 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:03.773612 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:03.773657 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:03.773959 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.773975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.775055 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.775066 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.776449 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.776461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.777621 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.777631 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.777866 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.777892 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 02:51:03.778019 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 02:51:03.778167 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 02:51:03.778487 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.772785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41495 (local address 127.0.97.2:34623)
0929 02:51:03.772987 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.772992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.773069 (+ 77us) server_negotiation.cc:407] Connection header received
0929 02:51:03.773298 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.773307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.773411 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.773523 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:03.774154 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.774880 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.776659 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.777455 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.777612 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.777725 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.778045 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.778181 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:51:03.779877 19647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.773232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:33647 (local address 127.0.97.3:46191)
0929 02:51:03.774133 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:03.774148 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:03.774306 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:03.774670 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:03.774675 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:03.774701 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:03.775061 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.775070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.776147 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.776153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.777688 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:03.777699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.779104 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.779108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:03.779294 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.779334 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 02:51:03.779451 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 02:51:03.779578 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":578,"threads_started":1}
I20260929 02:51:03.779738 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b7e359b224e51b840c4e58d9878df" candidate_uuid: "47bc14196891498eb8444d97b88414c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f5a6b245534d608138c2096453d6ac" is_pre_election: true
I20260929 02:51:03.780004 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.773514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46191 (local address 127.0.97.1:33647)
0929 02:51:03.773724 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.773732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.774244 (+ 512us) server_negotiation.cc:407] Connection header received
0929 02:51:03.774455 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.774463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.774539 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.774644 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:03.775226 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.775988 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.777885 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.778914 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.779062 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.779172 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.779518 (+ 346us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.779686 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 02:51:03.780296 19445 raft_consensus.cc:2454] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47bc14196891498eb8444d97b88414c4 in term 0.
I20260929 02:51:03.781065 19509 leader_election.cc:302] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47bc14196891498eb8444d97b88414c4, 71f5a6b245534d608138c2096453d6ac; no voters:
I20260929 02:51:03.781035 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b7e359b224e51b840c4e58d9878df" candidate_uuid: "47bc14196891498eb8444d97b88414c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" is_pre_election: true
I20260929 02:51:03.781530 19645 raft_consensus.cc:2781] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:51:03.781543 19314 raft_consensus.cc:2454] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47bc14196891498eb8444d97b88414c4 in term 0.
I20260929 02:51:03.781658 19645 raft_consensus.cc:482] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:51:03.781808 19645 raft_consensus.cc:3037] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:03.784336 19645 raft_consensus.cc:504] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.785255 19645 leader_election.cc:288] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [CANDIDATE]: Term 1 election: Requested vote from peers 71f5a6b245534d608138c2096453d6ac (127.0.97.2:34623), 33a5db07002c4354bf3e5c92d4b9e4c5 (127.0.97.1:33647)
I20260929 02:51:03.785708 19445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b7e359b224e51b840c4e58d9878df" candidate_uuid: "47bc14196891498eb8444d97b88414c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71f5a6b245534d608138c2096453d6ac"
I20260929 02:51:03.785951 19445 raft_consensus.cc:3037] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:03.786195 19314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "997b7e359b224e51b840c4e58d9878df" candidate_uuid: "47bc14196891498eb8444d97b88414c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5"
I20260929 02:51:03.786424 19314 raft_consensus.cc:3037] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:03.789419 19445 raft_consensus.cc:2454] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47bc14196891498eb8444d97b88414c4 in term 1.
I20260929 02:51:03.789644 19314 raft_consensus.cc:2454] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47bc14196891498eb8444d97b88414c4 in term 1.
I20260929 02:51:03.790005 19509 leader_election.cc:302] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47bc14196891498eb8444d97b88414c4, 71f5a6b245534d608138c2096453d6ac; no voters:
I20260929 02:51:03.790301 19645 raft_consensus.cc:2781] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:03.790752 19645 raft_consensus.cc:686] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 1 LEADER]: Becoming Leader. State: Replica: 47bc14196891498eb8444d97b88414c4, State: Running, Role: LEADER
I20260929 02:51:03.791307 19645 consensus_queue.cc:237] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } }
I20260929 02:51:03.796224 19173 catalog_manager.cc:5495] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47bc14196891498eb8444d97b88414c4 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "47bc14196891498eb8444d97b88414c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47bc14196891498eb8444d97b88414c4" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 } health_report { overall_health: UNKNOWN } } }
I20260929 02:51:03.819928 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.813850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.0.97.3:37957)
0929 02:51:03.814112 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.814117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.814166 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:51:03.814390 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.814393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.814478 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.814615 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:03.815139 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.815895 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.817583 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.818003 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.818139 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.818708 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:03.819375 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:03.819515 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.819687 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.819729 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:51:03.828670 19314 raft_consensus.cc:1261] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 1 FOLLOWER]: Refusing update from remote peer 47bc14196891498eb8444d97b88414c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:03.828684 19445 raft_consensus.cc:1261] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 1 FOLLOWER]: Refusing update from remote peer 47bc14196891498eb8444d97b88414c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:03.829739 19648 consensus_queue.cc:1035] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71f5a6b245534d608138c2096453d6ac" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:03.830210 19645 consensus_queue.cc:1035] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33a5db07002c4354bf3e5c92d4b9e4c5" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 33647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:03.841679 19651 mvcc.cc:205] Tried to move back new op lower bound from 7334503480631447552 to 7334503480492929024. Current Snapshot: MvccSnapshot[applied={T|T < 7334503480631447552}]
I20260929 02:51:03.856447 19173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:03.856858 19173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33536:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:03.861266 19173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fc1e487d451843bb87dae58fdf90f911: Sending DeleteTablet for 3 replicas of tablet 997b7e359b224e51b840c4e58d9878df
I20260929 02:51:03.862747 19555 tablet_service.cc:1514] Processing DeleteTablet for tablet 997b7e359b224e51b840c4e58d9878df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:03 UTC) from {username='slave'} at 127.0.0.1:59582
I20260929 02:51:03.862759 19294 tablet_service.cc:1514] Processing DeleteTablet for tablet 997b7e359b224e51b840c4e58d9878df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:03 UTC) from {username='slave'} at 127.0.0.1:35814
I20260929 02:51:03.863770 19660 tablet_replica.cc:330] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: stopping tablet replica
I20260929 02:51:03.863875 19425 tablet_service.cc:1514] Processing DeleteTablet for tablet 997b7e359b224e51b840c4e58d9878df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:03 UTC) from {username='slave'} at 127.0.0.1:40296
I20260929 02:51:03.864279 19660 raft_consensus.cc:2229] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:03.864970 19660 raft_consensus.cc:2258] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:03.868024 19660 ts_tablet_manager.cc:1905] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:03.868920 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 19232
I20260929 02:51:03.875204 19662 tablet_replica.cc:330] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: stopping tablet replica
I20260929 02:51:03.875967 19662 raft_consensus.cc:2229] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:03.876682 19662 raft_consensus.cc:2258] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:03.879070 19662 ts_tablet_manager.cc:1905] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:03.880139 19660 ts_tablet_manager.cc:1918] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:03.880347 19660 log.cc:1196] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/997b7e359b224e51b840c4e58d9878df
I20260929 02:51:03.880981 19660 ts_tablet_manager.cc:1939] T 997b7e359b224e51b840c4e58d9878df P 33a5db07002c4354bf3e5c92d4b9e4c5: Deleting consensus metadata
I20260929 02:51:03.883239 19158 catalog_manager.cc:4841] TS 33a5db07002c4354bf3e5c92d4b9e4c5 (127.0.97.1:33647): tablet 997b7e359b224e51b840c4e58d9878df (table test-table [id=cd0fc62219cc479fa6223bae2b96c17f]) successfully deleted
I20260929 02:51:03.887019 19662 ts_tablet_manager.cc:1918] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:03.887234 19662 log.cc:1196] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/997b7e359b224e51b840c4e58d9878df
I20260929 02:51:03.887889 19662 ts_tablet_manager.cc:1939] T 997b7e359b224e51b840c4e58d9878df P 71f5a6b245534d608138c2096453d6ac: Deleting consensus metadata
I20260929 02:51:03.889607 19661 tablet_replica.cc:330] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: stopping tablet replica
I20260929 02:51:03.890225 19158 catalog_manager.cc:4841] TS 71f5a6b245534d608138c2096453d6ac (127.0.97.2:34623): tablet 997b7e359b224e51b840c4e58d9878df (table test-table [id=cd0fc62219cc479fa6223bae2b96c17f]) successfully deleted
I20260929 02:51:03.892287 19661 raft_consensus.cc:2229] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:51:03.893090 19661 raft_consensus.cc:2258] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:03.895680 19661 ts_tablet_manager.cc:1905] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:03.895993 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:03.870429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.0.97.1:33647)
0929 02:51:03.870638 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:03.870645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:03.887435 (+ 16790us) server_negotiation.cc:407] Connection header received
0929 02:51:03.887780 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:03.887786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:03.887913 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:03.888058 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:03.888515 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.889803 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.891846 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:03.892303 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:03.892485 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:03.893945 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:03.893966 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:03.893990 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:03.894955 (+ 965us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:03.895014 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:03.895038 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:03.895186 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:03.895346 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 02:51:03.895474 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:51:03.896804 19354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:03.904579 19661 ts_tablet_manager.cc:1918] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:03.904811 19661 log.cc:1196] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/997b7e359b224e51b840c4e58d9878df
I20260929 02:51:03.905414 19661 ts_tablet_manager.cc:1939] T 997b7e359b224e51b840c4e58d9878df P 47bc14196891498eb8444d97b88414c4: Deleting consensus metadata
I20260929 02:51:03.907441 19159 catalog_manager.cc:4841] TS 47bc14196891498eb8444d97b88414c4 (127.0.97.3:37957): tablet 997b7e359b224e51b840c4e58d9878df (table test-table [id=cd0fc62219cc479fa6223bae2b96c17f]) successfully deleted
I20260929 02:51:03.991567 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 19232
I20260929 02:51:04.016737 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 19363
I20260929 02:51:04.024966 19633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:04.017598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40310 (local address 127.0.97.2:34623)
0929 02:51:04.017783 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:04.017788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:04.017841 (+ 53us) server_negotiation.cc:407] Connection header received
0929 02:51:04.018073 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:04.018078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:04.018177 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:04.018266 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:04.018728 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:04.019724 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:04.021735 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:04.022149 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:04.022266 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:04.023453 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:04.023462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:04.023478 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:04.024091 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:04.024143 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:04.024163 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:04.024255 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:04.024417 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 02:51:04.024516 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:51:04.025673 19485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:04.128633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 19363
I20260929 02:51:04.151839 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 19493
I20260929 02:51:04.160319 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:04.152889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.0.97.3:37957)
0929 02:51:04.153075 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:04.153080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:04.153103 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:51:04.153219 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:04.153223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:04.153307 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:04.153406 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:04.153935 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:04.154798 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:04.156742 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:04.157287 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:04.157402 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:04.158644 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:04.158653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:04.158669 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:04.159488 (+ 819us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:04.159541 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:04.159561 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:04.159662 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:04.159845 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 02:51:04.159932 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 02:51:04.161293 19615 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:04.291356 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 19493
I20260929 02:51:04.310562 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 19142
I20260929 02:51:04.311853 19203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:04.423448 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 19142
2026-09-29T02:51:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2907 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:04 dist-test-slave-6tjh krb5kdc[19670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:04 dist-test-slave-6tjh krb5kdc[19670](info): set up 2 sockets
Sep 29 02:51:04 dist-test-slave-6tjh krb5kdc[19670](info): commencing operation
krb5kdc: starting...
W20260929 02:51:05.597267 388 mini_kdc.cc:121] Time spent starting KDC: real 1.132s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:05 dist-test-slave-6tjh krb5kdc[19670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:05.873934 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39353
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39353
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:06.224910 19683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:06.225243 19683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:06.225411 19683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:06.233088 19683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:06.233210 19683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:06.233302 19683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:06.233415 19683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:06.233511 19683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:06.243652 19683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:06.245066 19683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:06.246435 19683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:06.258646 19688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:06.258616 19689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:06.259128 19691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:06.259938 19683 server_base.cc:1034] running on GCE node
I20260929 02:51:06.260905 19683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:06.262514 19683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:06.263810 19683 hybrid_clock.cc:648] HybridClock initialized: now 1790650266263747 us; error 42 us; skew 500 ppm
I20260929 02:51:06.264292 19683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:06 dist-test-slave-6tjh krb5kdc[19670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:06.268306 19683 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:06.270309 19683 webserver.cc:460] Webserver started at http://127.0.97.62:36115/ using document root <none> and password file <none>
I20260929 02:51:06.271157 19683 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:06.271324 19683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:06.271698 19683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:06.275332 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "a5e1e6648f9447348a0b9f9fec20ab51"
format_stamp: "Formatted at 2026-09-29 02:51:06 on dist-test-slave-6tjh"
server_key: "da2d04b0cc837f8f285cee368ff2417f"
server_key_iv: "4037623c0b06a7344e1685f709510044"
server_key_version: "encryptionkey@0"
I20260929 02:51:06.276245 19683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "a5e1e6648f9447348a0b9f9fec20ab51"
format_stamp: "Formatted at 2026-09-29 02:51:06 on dist-test-slave-6tjh"
server_key: "da2d04b0cc837f8f285cee368ff2417f"
server_key_iv: "4037623c0b06a7344e1685f709510044"
server_key_version: "encryptionkey@0"
I20260929 02:51:06.282184 19683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 02:51:06.286425 19698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:06.288062 19683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:51:06.288307 19683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "a5e1e6648f9447348a0b9f9fec20ab51"
format_stamp: "Formatted at 2026-09-29 02:51:06 on dist-test-slave-6tjh"
server_key: "da2d04b0cc837f8f285cee368ff2417f"
server_key_iv: "4037623c0b06a7344e1685f709510044"
server_key_version: "encryptionkey@0"
I20260929 02:51:06.288646 19683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:06.312073 19683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:06.317096 19683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:06.317417 19683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:06.336984 19683 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39353
I20260929 02:51:06.337007 19749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39353 every 8 connection(s)
I20260929 02:51:06.338442 19683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:06.340003 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 19683
I20260929 02:51:06.340202 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:06.340724 388 external_mini_cluster.cc:1428] Setting key f0072e9ae6a955a50276c41ca5d86b55
I20260929 02:51:06.343041 19750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:51:06 dist-test-slave-6tjh krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:06.356827 19750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51: Bootstrap starting.
I20260929 02:51:06.361460 19750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:06.362836 19750 log.cc:824] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:06.366608 19750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51: No bootstrap required, opened a new log
I20260929 02:51:06.366919 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:06.342667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.0.97.62:39353)
0929 02:51:06.343502 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:06.343530 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:06.343585 (+ 55us) server_negotiation.cc:407] Connection header received
0929 02:51:06.345974 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:06.346005 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:06.346572 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:06.347051 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:06.348434 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:06.350539 (+ 2105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:06.351890 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:06.352993 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:06.353208 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:06.357733 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:06.357745 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:06.357765 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:06.357815 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:06.362065 (+ 4250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:06.362906 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:06.362915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:06.362924 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:06.363156 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:06.363632 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:06.363636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:06.363639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:06.364323 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:06.364648 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:06.365077 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 02:51:06.365540 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":371,"threads_started":1}
I20260929 02:51:06.375490 19750 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39353 } }
I20260929 02:51:06.375892 19750 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:06.376013 19750 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e1e6648f9447348a0b9f9fec20ab51, State: Initialized, Role: FOLLOWER
I20260929 02:51:06.376719 19750 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39353 } }
I20260929 02:51:06.376974 19750 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:06.377134 19750 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:06.377328 19750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:06.379534 19750 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39353 } }
I20260929 02:51:06.380090 19750 leader_election.cc:302] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5e1e6648f9447348a0b9f9fec20ab51; no voters:
I20260929 02:51:06.380817 19750 leader_election.cc:288] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:06.381003 19755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:06.382023 19755 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [term 1 LEADER]: Becoming Leader. State: Replica: a5e1e6648f9447348a0b9f9fec20ab51, State: Running, Role: LEADER
I20260929 02:51:06.382587 19755 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39353 } }
I20260929 02:51:06.383136 19750 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:06.385766 19757 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5e1e6648f9447348a0b9f9fec20ab51. Latest consensus state: current_term: 1 leader_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39353 } } }
I20260929 02:51:06.386164 19757 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:06.385699 19756 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e1e6648f9447348a0b9f9fec20ab51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39353 } } }
I20260929 02:51:06.386370 19756 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:06.387223 19763 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:06.396795 19763 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:06.407043 19763 catalog_manager.cc:1347] Generated new cluster ID: ee6ac0583b88451ab0b5b2f5e094325c
I20260929 02:51:06.407187 19763 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:06.421149 19763 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:06.422662 19763 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:06.430627 19763 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51: Generated new TSK 0
I20260929 02:51:06.431602 19763 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260929 02:51:06.508344 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39353
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:06.858199 19776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:06.858517 19776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:06.858677 19776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:06.867093 19776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:06.867219 19776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:06.867403 19776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:51:06.878060 19776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:06.879503 19776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:06.880991 19776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:06.895126 19784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:06.895200 19782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:06.895180 19781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:06.896224 19776 server_base.cc:1034] running on GCE node
I20260929 02:51:06.896976 19776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:06.898063 19776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:06.899293 19776 hybrid_clock.cc:648] HybridClock initialized: now 1790650266899240 us; error 43 us; skew 500 ppm
I20260929 02:51:06.899648 19776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:06 dist-test-slave-6tjh krb5kdc[19670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:06.903441 19776 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260929 02:51:06.905185 19776 webserver.cc:460] Webserver started at http://127.0.97.1:41143/ using document root <none> and password file <none>
I20260929 02:51:06.905947 19776 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:06.906082 19776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:06.906417 19776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:06.909536 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "442c9269865748c9a3a5d47b46fc931e"
format_stamp: "Formatted at 2026-09-29 02:51:06 on dist-test-slave-6tjh"
server_key: "2d6542c2b3c2b8ab97cf5986768fe6f1"
server_key_iv: "f90377e10c81610caf26b62eb105f8a1"
server_key_version: "encryptionkey@0"
I20260929 02:51:06.910413 19776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "442c9269865748c9a3a5d47b46fc931e"
format_stamp: "Formatted at 2026-09-29 02:51:06 on dist-test-slave-6tjh"
server_key: "2d6542c2b3c2b8ab97cf5986768fe6f1"
server_key_iv: "f90377e10c81610caf26b62eb105f8a1"
server_key_version: "encryptionkey@0"
I20260929 02:51:06.915839 19776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:51:06.919684 19791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:06.921217 19776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:06.921392 19776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "442c9269865748c9a3a5d47b46fc931e"
format_stamp: "Formatted at 2026-09-29 02:51:06 on dist-test-slave-6tjh"
server_key: "2d6542c2b3c2b8ab97cf5986768fe6f1"
server_key_iv: "f90377e10c81610caf26b62eb105f8a1"
server_key_version: "encryptionkey@0"
I20260929 02:51:06.921624 19776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:06.946120 19776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:06.950783 19776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:06.951113 19776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:06.952172 19776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:06.953613 19776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:06.953686 19776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:06.953816 19776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:06.953879 19776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:06.988832 19776 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37243
I20260929 02:51:06.988883 19903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37243 every 8 connection(s)
I20260929 02:51:06.990208 19776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:51:06.998206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 19776
I20260929 02:51:06.998391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:51:06.998837 388 external_mini_cluster.cc:1428] Setting key 074f68e899e89281bde573ac5ca5ccdb
Sep 29 02:51:06 dist-test-slave-6tjh krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:07.011507 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:06.992942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39935 (local address 127.0.97.62:39353)
0929 02:51:06.993472 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:06.993477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:06.994583 (+ 1106us) server_negotiation.cc:407] Connection header received
0929 02:51:06.996478 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:06.996484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:06.996620 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:06.996742 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:06.998583 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:06.999387 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:07.001389 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:07.001803 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:07.001945 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:07.005577 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:07.005582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:07.005585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:07.005618 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:07.008269 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:07.009114 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:07.009120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:07.009124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:07.009251 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:07.009726 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:07.009731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:07.009736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:07.010014 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:07.010171 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:07.010894 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 02:51:07.011128 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":221,"threads_started":1}
I20260929 02:51:07.012534 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:06.993665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39353 (local address 127.0.97.1:39935)
0929 02:51:06.994403 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:06.994460 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:06.996201 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:06.996985 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:06.996995 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:06.997532 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:06.998336 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:06.998364 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:06.999578 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:06.999591 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:07.001224 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:07.001237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:07.002052 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:07.002061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:07.002312 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:07.003024 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:07.003049 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:07.005324 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:07.008434 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:07.008444 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:07.008463 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:07.008986 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:07.009368 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:07.009373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:07.009377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:07.009592 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:07.010152 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:07.010160 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:07.010509 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 02:51:07.010781 (+ 272us) client_negotiation.cc:240] Negotiation successful
0929 02:51:07.011358 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":326,"threads_started":1}
I20260929 02:51:07.014340 19904 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39353
I20260929 02:51:07.014689 19904 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:07.015560 19904 heartbeater.cc:511] Master 127.0.97.62:39353 requested a full tablet report, sending...
I20260929 02:51:07.018124 19715 ts_manager.cc:194] Registered new tserver with Master: 442c9269865748c9a3a5d47b46fc931e (127.0.97.1:37243)
I20260929 02:51:07.020403 19715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39935
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260929 02:51:07.056907 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39353
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:07.397534 19910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:07.397892 19910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:07.398061 19910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:07.405573 19910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:07.405736 19910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:07.405908 19910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:51:07.416257 19910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:07.417708 19910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:07.419234 19910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:07.433435 19915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:07.433531 19916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:07.433534 19918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:07.434535 19910 server_base.cc:1034] running on GCE node
I20260929 02:51:07.435295 19910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:07.436425 19910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:07.437704 19910 hybrid_clock.cc:648] HybridClock initialized: now 1790650267437646 us; error 57 us; skew 500 ppm
I20260929 02:51:07.438119 19910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:07 dist-test-slave-6tjh krb5kdc[19670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:07.442025 19910 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260929 02:51:07.443768 19910 webserver.cc:460] Webserver started at http://127.0.97.2:37055/ using document root <none> and password file <none>
I20260929 02:51:07.444643 19910 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:07.444810 19910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:07.445164 19910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:07.448319 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "4e9b2fa97e0b4dc688b6e91629293b9c"
format_stamp: "Formatted at 2026-09-29 02:51:07 on dist-test-slave-6tjh"
server_key: "27518fff6811ad29c5c897f7ccdaf2a3"
server_key_iv: "cb3f4acaee0ece98d738bd0be946d4c3"
server_key_version: "encryptionkey@0"
I20260929 02:51:07.449285 19910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4e9b2fa97e0b4dc688b6e91629293b9c"
format_stamp: "Formatted at 2026-09-29 02:51:07 on dist-test-slave-6tjh"
server_key: "27518fff6811ad29c5c897f7ccdaf2a3"
server_key_iv: "cb3f4acaee0ece98d738bd0be946d4c3"
server_key_version: "encryptionkey@0"
I20260929 02:51:07.455170 19910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:51:07.459383 19925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:07.461184 19910 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 02:51:07.461391 19910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "4e9b2fa97e0b4dc688b6e91629293b9c"
format_stamp: "Formatted at 2026-09-29 02:51:07 on dist-test-slave-6tjh"
server_key: "27518fff6811ad29c5c897f7ccdaf2a3"
server_key_iv: "cb3f4acaee0ece98d738bd0be946d4c3"
server_key_version: "encryptionkey@0"
I20260929 02:51:07.461633 19910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:07.477336 19910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:07.481940 19910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:07.482250 19910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:07.483348 19910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:07.484849 19910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:07.484954 19910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:07.485088 19910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:07.485178 19910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:07.520026 19910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44655
I20260929 02:51:07.520118 20037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44655 every 8 connection(s)
I20260929 02:51:07.521449 19910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:51:07.522845 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 19910
I20260929 02:51:07.523051 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:51:07.523617 388 external_mini_cluster.cc:1428] Setting key 0d7ba5d5423b8703efe2bddde6f0d889
Sep 29 02:51:07 dist-test-slave-6tjh krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:07.548697 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:07.525029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42725 (local address 127.0.97.62:39353)
0929 02:51:07.526027 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:07.526033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:07.526801 (+ 768us) server_negotiation.cc:407] Connection header received
0929 02:51:07.529984 (+ 3183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:07.529990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:07.530087 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:07.530200 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:07.532651 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:07.533428 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:07.536226 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:07.537094 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:07.537259 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:07.541053 (+ 3794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:07.541063 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:07.541070 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:07.541120 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:07.545031 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:07.545870 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:07.545878 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:07.545885 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:07.546071 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:07.546524 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:07.546531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:07.546537 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:07.546955 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:07.547112 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:07.547841 (+ 729us) server_negotiation.cc:299] Negotiation successful
0929 02:51:07.548134 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":867,"thread_start_us":769,"threads_started":1}
I20260929 02:51:07.549443 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:07.525704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39353 (local address 127.0.97.2:42725)
0929 02:51:07.526608 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:07.526670 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:07.529532 (+ 2862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:07.530507 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:07.530523 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:07.531382 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:07.532314 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:07.532345 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:07.533622 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:07.533631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:07.536007 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:07.536031 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:07.537361 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:07.537366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:07.537595 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:07.538381 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:07.538406 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:07.540779 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:07.545212 (+ 4433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:07.545220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:07.545236 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:07.545727 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:07.546195 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:07.546198 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:07.546200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:07.546402 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:07.547092 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:07.547098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:07.547442 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 02:51:07.547696 (+ 254us) client_negotiation.cc:240] Negotiation successful
0929 02:51:07.548147 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":392,"threads_started":1}
I20260929 02:51:07.551115 20038 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39353
I20260929 02:51:07.551466 20038 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:07.552220 20038 heartbeater.cc:511] Master 127.0.97.62:39353 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260929 02:51:07.553905 19715 ts_manager.cc:194] Registered new tserver with Master: 4e9b2fa97e0b4dc688b6e91629293b9c (127.0.97.2:44655)
I20260929 02:51:07.554847 19715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:42725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260929 02:51:07.589409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39353
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:07.939198 20044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:07.939540 20044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:07.939728 20044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:07.948127 20044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:07.948267 20044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:07.948477 20044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:51:07.960006 20044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39353
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:07.961462 20044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:07.963022 20044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:07.977669 20050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:07.977696 20052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:07.977619 20049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:07.978528 20044 server_base.cc:1034] running on GCE node
I20260929 02:51:07.979216 20044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:07.980309 20044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:07.981539 20044 hybrid_clock.cc:648] HybridClock initialized: now 1790650267981489 us; error 82 us; skew 500 ppm
I20260929 02:51:07.981918 20044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:07 dist-test-slave-6tjh krb5kdc[19670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:07.985885 20044 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260929 02:51:07.987681 20044 webserver.cc:460] Webserver started at http://127.0.97.3:36129/ using document root <none> and password file <none>
I20260929 02:51:07.988631 20044 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:07.988791 20044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:07.989130 20044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:07.992304 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "c62e8087afe049d78e6d5aa76ba2dbd1"
format_stamp: "Formatted at 2026-09-29 02:51:07 on dist-test-slave-6tjh"
server_key: "a79a576cfe52db82ff8bf3973d6bee63"
server_key_iv: "ae619f1a4bb69414726d924d17517f0d"
server_key_version: "encryptionkey@0"
I20260929 02:51:07.993294 20044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "c62e8087afe049d78e6d5aa76ba2dbd1"
format_stamp: "Formatted at 2026-09-29 02:51:07 on dist-test-slave-6tjh"
server_key: "a79a576cfe52db82ff8bf3973d6bee63"
server_key_iv: "ae619f1a4bb69414726d924d17517f0d"
server_key_version: "encryptionkey@0"
I20260929 02:51:07.999054 20044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:51:08.002875 20059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:08.004637 20044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:08.004837 20044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "c62e8087afe049d78e6d5aa76ba2dbd1"
format_stamp: "Formatted at 2026-09-29 02:51:07 on dist-test-slave-6tjh"
server_key: "a79a576cfe52db82ff8bf3973d6bee63"
server_key_iv: "ae619f1a4bb69414726d924d17517f0d"
server_key_version: "encryptionkey@0"
I20260929 02:51:08.005069 20044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:08.018458 20044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:08.022977 20044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:08.023295 20044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:08.024403 20044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:08.024868 19904 heartbeater.cc:503] Master 127.0.97.62:39353 was elected leader, sending a full tablet report...
I20260929 02:51:08.026197 20044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:08.026304 20044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:08.026491 20044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:08.026563 20044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:08.060845 20044 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42691
I20260929 02:51:08.060894 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42691 every 8 connection(s)
I20260929 02:51:08.062341 20044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:51:08.065798 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20044
I20260929 02:51:08.066004 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:51:08.066695 388 external_mini_cluster.cc:1428] Setting key 8db07d46d478f1a8d5a1d9bd1741c449
Sep 29 02:51:08 dist-test-slave-6tjh krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:08.086016 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.065258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42861 (local address 127.0.97.62:39353)
0929 02:51:08.065784 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.065791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.067074 (+ 1283us) server_negotiation.cc:407] Connection header received
0929 02:51:08.069585 (+ 2511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.069591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.069709 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.069833 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:08.071438 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.072262 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.074245 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.074679 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.074809 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.079037 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:08.079043 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:08.079047 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:08.079087 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:08.082721 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.083666 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.083671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.083673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.083813 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.084267 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.084272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.084274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.084675 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:08.084793 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.085544 (+ 751us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.085730 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":260,"threads_started":1}
I20260929 02:51:08.087061 20175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.065847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39353 (local address 127.0.97.3:42861)
0929 02:51:08.066868 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:08.066922 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:08.069266 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:08.070080 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:08.070089 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:08.070664 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:08.071262 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.071279 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.072414 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.072418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.074013 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.074033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.074925 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.074930 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.075184 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.075909 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:08.075936 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:08.078709 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:08.082900 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:08.082908 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:08.082924 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:08.083502 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:08.083912 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:08.083915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:08.083917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:08.084139 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:08.084791 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:08.084798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:08.085178 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 02:51:08.085419 (+ 241us) client_negotiation.cc:240] Negotiation successful
0929 02:51:08.085848 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":388,"threads_started":1}
I20260929 02:51:08.088653 20172 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39353
I20260929 02:51:08.089032 20172 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:08.089794 20172 heartbeater.cc:511] Master 127.0.97.62:39353 requested a full tablet report, sending...
I20260929 02:51:08.091382 19715 ts_manager.cc:194] Registered new tserver with Master: c62e8087afe049d78e6d5aa76ba2dbd1 (127.0.97.3:42691)
I20260929 02:51:08.092245 19715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:42861
I20260929 02:51:08.095568 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:51:08.115224 20174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.100221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.0.97.62:39353)
0929 02:51:08.100492 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.100501 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.100905 (+ 404us) server_negotiation.cc:407] Connection header received
0929 02:51:08.101144 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.101149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.101219 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.101336 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:08.103226 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.104360 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.105975 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.106472 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.106623 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.109393 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:08.109399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:08.109402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:08.109443 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:08.112256 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.113302 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.113306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.113308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.113446 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.113822 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.113825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.113828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.114186 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:08.114373 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.114759 (+ 386us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.114982 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 02:51:08.119418 19715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:51:08.126464 19715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:51:08.173848 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.140741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44288 (local address 127.0.97.3:42691)
0929 02:51:08.161369 (+ 20628us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.161382 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.161432 (+ 50us) server_negotiation.cc:407] Connection header received
0929 02:51:08.161585 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.161598 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.161962 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.162264 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:08.163333 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.165461 (+ 2128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.168452 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.170155 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.170297 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.170550 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.173277 (+ 2727us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.173485 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20408,"thread_start_us":20241,"threads_started":1}
I20260929 02:51:08.174228 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.140399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42691 (local address 127.0.0.1:44288)
0929 02:51:08.157510 (+ 17111us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:08.157593 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:08.157918 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:08.162357 (+ 4439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:08.162371 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:08.162431 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:08.163014 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.163036 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.165649 (+ 2613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.165655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.168259 (+ 2604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.168277 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.171908 (+ 3631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.171914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.172178 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.172339 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 02:51:08.172503 (+ 164us) client_negotiation.cc:240] Negotiation successful
0929 02:51:08.172754 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16027,"mutex_wait_us":36,"thread_start_us":15809,"threads_started":1}
I20260929 02:51:08.174988 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.141096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:37243 (local address 127.0.0.1:57286)
0929 02:51:08.165431 (+ 24335us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:08.165457 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:08.165627 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:08.167008 (+ 1381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:08.167017 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:08.167061 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:08.167488 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.167497 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.170301 (+ 2804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.170308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.171624 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.171637 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.174166 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.174174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.174384 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.174423 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 02:51:08.174497 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 02:51:08.174665 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24167,"thread_start_us":26011,"threads_started":1}
I20260929 02:51:08.175014 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.141445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.0.97.1:37243)
0929 02:51:08.160508 (+ 19063us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.160522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.165622 (+ 5100us) server_negotiation.cc:407] Connection header received
0929 02:51:08.165805 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.165819 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.166210 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.166571 (+ 361us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:08.167678 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.170161 (+ 2483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.171801 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.173833 (+ 2032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.173995 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.174266 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.174584 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.174758 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18786,"thread_start_us":19092,"threads_started":1}
I20260929 02:51:08.178571 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.141067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.0.97.2:44655)
0929 02:51:08.166909 (+ 25842us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.166919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.168056 (+ 1137us) server_negotiation.cc:407] Connection header received
0929 02:51:08.168242 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.168248 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.168504 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.168742 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:08.169660 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.171660 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.175292 (+ 3632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.177146 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.177283 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.177466 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.178030 (+ 564us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.178217 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25657,"thread_start_us":26068,"threads_started":1}
I20260929 02:51:08.178619 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.141391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44655 (local address 127.0.0.1:36502)
0929 02:51:08.167929 (+ 26538us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:08.167945 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:08.168088 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:08.168987 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:08.168993 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:08.169017 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:08.169434 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.169446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.171811 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.171818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.173840 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.173853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.177628 (+ 3775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.177635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.177810 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.177845 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:51:08.177954 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 02:51:08.178098 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26411,"mutex_wait_us":47,"thread_start_us":24334,"threads_started":1}
I20260929 02:51:08.178962 19839 tablet_service.cc:1467] Processing CreateTablet for tablet 73ac6adab2d94ac09e47feb30e2f1784 (DEFAULT_TABLE table=test-table [id=a827b36bb6f14aa7a0608b15ff6235a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:08.179152 20107 tablet_service.cc:1467] Processing CreateTablet for tablet 73ac6adab2d94ac09e47feb30e2f1784 (DEFAULT_TABLE table=test-table [id=a827b36bb6f14aa7a0608b15ff6235a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:08.180222 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ac6adab2d94ac09e47feb30e2f1784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:08.180222 19839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ac6adab2d94ac09e47feb30e2f1784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:08.184762 19973 tablet_service.cc:1467] Processing CreateTablet for tablet 73ac6adab2d94ac09e47feb30e2f1784 (DEFAULT_TABLE table=test-table [id=a827b36bb6f14aa7a0608b15ff6235a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:08.186720 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ac6adab2d94ac09e47feb30e2f1784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:08.193564 20191 tablet_bootstrap.cc:492] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Bootstrap starting.
I20260929 02:51:08.196807 20191 tablet_bootstrap.cc:654] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:08.198029 20191 log.cc:824] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:08.199640 20192 tablet_bootstrap.cc:492] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Bootstrap starting.
I20260929 02:51:08.202666 20191 tablet_bootstrap.cc:492] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: No bootstrap required, opened a new log
I20260929 02:51:08.202702 20192 tablet_bootstrap.cc:654] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:08.202998 20191 ts_tablet_manager.cc:1397] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260929 02:51:08.203653 20194 tablet_bootstrap.cc:492] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Bootstrap starting.
I20260929 02:51:08.204090 20192 log.cc:824] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:08.207476 20194 tablet_bootstrap.cc:654] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:08.207587 20192 tablet_bootstrap.cc:492] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: No bootstrap required, opened a new log
I20260929 02:51:08.207870 20192 ts_tablet_manager.cc:1397] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 02:51:08.209321 20194 log.cc:824] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:08.212400 20191 raft_consensus.cc:348] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.212775 20194 tablet_bootstrap.cc:492] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: No bootstrap required, opened a new log
I20260929 02:51:08.212904 20191 raft_consensus.cc:374] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:08.213004 20191 raft_consensus.cc:729] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62e8087afe049d78e6d5aa76ba2dbd1, State: Initialized, Role: FOLLOWER
I20260929 02:51:08.213068 20194 ts_tablet_manager.cc:1397] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 02:51:08.213671 20191 consensus_queue.cc:260] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.214846 20172 heartbeater.cc:503] Master 127.0.97.62:39353 was elected leader, sending a full tablet report...
I20260929 02:51:08.216846 20191 ts_tablet_manager.cc:1428] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Time spent starting tablet: real 0.014s user 0.007s sys 0.008s
I20260929 02:51:08.218703 20192 raft_consensus.cc:348] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.219165 20192 raft_consensus.cc:374] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:08.219266 20192 raft_consensus.cc:729] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 442c9269865748c9a3a5d47b46fc931e, State: Initialized, Role: FOLLOWER
I20260929 02:51:08.219909 20192 consensus_queue.cc:260] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.221879 20192 ts_tablet_manager.cc:1428] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260929 02:51:08.223261 20194 raft_consensus.cc:348] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.223608 20194 raft_consensus.cc:374] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:08.223676 20194 raft_consensus.cc:729] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e9b2fa97e0b4dc688b6e91629293b9c, State: Initialized, Role: FOLLOWER
I20260929 02:51:08.224174 20194 consensus_queue.cc:260] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.225266 20038 heartbeater.cc:503] Master 127.0.97.62:39353 was elected leader, sending a full tablet report...
I20260929 02:51:08.225952 20194 ts_tablet_manager.cc:1428] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260929 02:51:08.242796 19905 tablet.cc:2369] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:08.243284 20198 raft_consensus.cc:482] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:51:08.243606 20198 raft_consensus.cc:504] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.245461 20198 leader_election.cc:288] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c62e8087afe049d78e6d5aa76ba2dbd1 (127.0.97.3:42691), 4e9b2fa97e0b4dc688b6e91629293b9c (127.0.97.2:44655)
I20260929 02:51:08.252718 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.245771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43887 (local address 127.0.97.3:42691)
0929 02:51:08.246005 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.246016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.246713 (+ 697us) server_negotiation.cc:407] Connection header received
0929 02:51:08.246959 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.246964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.247026 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.247123 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:08.247655 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.248436 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.250685 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.251558 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.251661 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.251741 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.252268 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.252404 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 02:51:08.253024 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.246083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39913 (local address 127.0.97.2:44655)
0929 02:51:08.246345 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.246354 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.247009 (+ 655us) server_negotiation.cc:407] Connection header received
0929 02:51:08.247230 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.247234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.247293 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.247397 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:08.247940 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.248956 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.250932 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.251921 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.252083 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.252191 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.252584 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.252727 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:51:08.253159 20201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.245883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44655 (local address 127.0.97.1:39913)
0929 02:51:08.246891 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:08.246905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:08.247076 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:08.247415 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:08.247419 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:08.247446 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:08.247778 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.247786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.249107 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.249116 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.250743 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.250758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.252191 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.252200 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.252375 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.252405 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 02:51:08.252504 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:51:08.252655 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":1,"thread_start_us":667,"threads_started":1}
I20260929 02:51:08.253223 20200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.245687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42691 (local address 127.0.97.1:43887)
0929 02:51:08.246588 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:08.246605 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:08.246805 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:08.247150 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:08.247155 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:08.247182 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:08.247507 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.247517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.248601 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.248606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.250522 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:08.250535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.251801 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.251809 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:08.252060 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.252100 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 02:51:08.252210 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 02:51:08.252337 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":69,"thread_start_us":577,"threads_started":1}
I20260929 02:51:08.254261 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac6adab2d94ac09e47feb30e2f1784" candidate_uuid: "442c9269865748c9a3a5d47b46fc931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" is_pre_election: true
I20260929 02:51:08.254297 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac6adab2d94ac09e47feb30e2f1784" candidate_uuid: "442c9269865748c9a3a5d47b46fc931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" is_pre_election: true
I20260929 02:51:08.254771 20127 raft_consensus.cc:2454] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442c9269865748c9a3a5d47b46fc931e in term 0.
I20260929 02:51:08.254797 19993 raft_consensus.cc:2454] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 442c9269865748c9a3a5d47b46fc931e in term 0.
I20260929 02:51:08.255456 19795 leader_election.cc:302] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442c9269865748c9a3a5d47b46fc931e, c62e8087afe049d78e6d5aa76ba2dbd1; no voters:
I20260929 02:51:08.255882 20198 raft_consensus.cc:2781] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:51:08.256032 20198 raft_consensus.cc:482] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:51:08.256151 20198 raft_consensus.cc:3037] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:08.258607 20198 raft_consensus.cc:504] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.259297 20198 leader_election.cc:288] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [CANDIDATE]: Term 1 election: Requested vote from peers c62e8087afe049d78e6d5aa76ba2dbd1 (127.0.97.3:42691), 4e9b2fa97e0b4dc688b6e91629293b9c (127.0.97.2:44655)
I20260929 02:51:08.259819 20127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac6adab2d94ac09e47feb30e2f1784" candidate_uuid: "442c9269865748c9a3a5d47b46fc931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1"
I20260929 02:51:08.259864 19993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ac6adab2d94ac09e47feb30e2f1784" candidate_uuid: "442c9269865748c9a3a5d47b46fc931e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c"
I20260929 02:51:08.260084 20127 raft_consensus.cc:3037] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:08.260097 19993 raft_consensus.cc:3037] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:08.262411 19993 raft_consensus.cc:2454] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442c9269865748c9a3a5d47b46fc931e in term 1.
I20260929 02:51:08.262409 20127 raft_consensus.cc:2454] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 442c9269865748c9a3a5d47b46fc931e in term 1.
I20260929 02:51:08.262902 19794 leader_election.cc:302] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 442c9269865748c9a3a5d47b46fc931e, 4e9b2fa97e0b4dc688b6e91629293b9c; no voters:
I20260929 02:51:08.263262 20198 raft_consensus.cc:2781] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:08.263739 20198 raft_consensus.cc:686] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 1 LEADER]: Becoming Leader. State: Replica: 442c9269865748c9a3a5d47b46fc931e, State: Running, Role: LEADER
I20260929 02:51:08.264535 20198 consensus_queue.cc:237] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } }
I20260929 02:51:08.270093 19715 catalog_manager.cc:5495] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e reported cstate change: term changed from 0 to 1, leader changed from <none> to 442c9269865748c9a3a5d47b46fc931e (127.0.97.1). New cstate: current_term: 1 leader_uuid: "442c9269865748c9a3a5d47b46fc931e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "442c9269865748c9a3a5d47b46fc931e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 37243 } health_report { overall_health: HEALTHY } } }
W20260929 02:51:08.274012 20039 tablet.cc:2369] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:08.307708 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.301095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.0.97.1:37243)
0929 02:51:08.301348 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.301355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.301474 (+ 119us) server_negotiation.cc:407] Connection header received
0929 02:51:08.301753 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.301757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.301851 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.302034 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:08.302448 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.303254 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.305020 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.305537 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.305669 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.306321 (+ 652us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:08.306868 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:08.307023 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.307244 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.307330 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 02:51:08.314069 20173 tablet.cc:2369] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:08.317154 20127 raft_consensus.cc:1261] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 1 FOLLOWER]: Refusing update from remote peer 442c9269865748c9a3a5d47b46fc931e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:08.317480 19993 raft_consensus.cc:1261] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 1 FOLLOWER]: Refusing update from remote peer 442c9269865748c9a3a5d47b46fc931e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:08.318382 20202 consensus_queue.cc:1035] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c62e8087afe049d78e6d5aa76ba2dbd1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:08.318768 20198 consensus_queue.cc:1035] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e9b2fa97e0b4dc688b6e91629293b9c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:08.353806 19715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:08.354099 19715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:08.357959 19715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5e1e6648f9447348a0b9f9fec20ab51: Sending DeleteTablet for 3 replicas of tablet 73ac6adab2d94ac09e47feb30e2f1784
I20260929 02:51:08.359367 20107 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ac6adab2d94ac09e47feb30e2f1784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44288
I20260929 02:51:08.359884 19973 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ac6adab2d94ac09e47feb30e2f1784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36502
I20260929 02:51:08.360524 20215 tablet_replica.cc:330] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: stopping tablet replica
I20260929 02:51:08.360855 20216 tablet_replica.cc:330] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: stopping tablet replica
I20260929 02:51:08.361155 20215 raft_consensus.cc:2229] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:08.361282 20216 raft_consensus.cc:2229] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:08.361699 19839 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ac6adab2d94ac09e47feb30e2f1784 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:08 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57286
I20260929 02:51:08.361845 20216 raft_consensus.cc:2258] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:08.361833 20215 raft_consensus.cc:2258] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:08.363287 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 19776
I20260929 02:51:08.364389 20215 ts_tablet_manager.cc:1905] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:08.364529 20216 ts_tablet_manager.cc:1905] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:08.372219 20215 ts_tablet_manager.cc:1918] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:08.372587 20215 log.cc:1196] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/73ac6adab2d94ac09e47feb30e2f1784
I20260929 02:51:08.373258 20215 ts_tablet_manager.cc:1939] T 73ac6adab2d94ac09e47feb30e2f1784 P c62e8087afe049d78e6d5aa76ba2dbd1: Deleting consensus metadata
I20260929 02:51:08.373404 20216 ts_tablet_manager.cc:1918] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:08.373610 20216 log.cc:1196] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/73ac6adab2d94ac09e47feb30e2f1784
I20260929 02:51:08.374260 20216 ts_tablet_manager.cc:1939] T 73ac6adab2d94ac09e47feb30e2f1784 P 4e9b2fa97e0b4dc688b6e91629293b9c: Deleting consensus metadata
I20260929 02:51:08.375756 19702 catalog_manager.cc:4841] TS c62e8087afe049d78e6d5aa76ba2dbd1 (127.0.97.3:42691): tablet 73ac6adab2d94ac09e47feb30e2f1784 (table test-table [id=a827b36bb6f14aa7a0608b15ff6235a7]) successfully deleted
I20260929 02:51:08.376228 19701 catalog_manager.cc:4841] TS 4e9b2fa97e0b4dc688b6e91629293b9c (127.0.97.2:44655): tablet 73ac6adab2d94ac09e47feb30e2f1784 (table test-table [id=a827b36bb6f14aa7a0608b15ff6235a7]) successfully deleted
I20260929 02:51:08.377866 20217 tablet_replica.cc:330] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: stopping tablet replica
I20260929 02:51:08.378394 20217 raft_consensus.cc:2229] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:51:08.379213 20217 raft_consensus.cc:2258] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:08.381705 20217 ts_tablet_manager.cc:1905] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 02:51:08 dist-test-slave-6tjh krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260929 02:51:08.390165 20217 ts_tablet_manager.cc:1918] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:08.390372 20217 log.cc:1196] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/73ac6adab2d94ac09e47feb30e2f1784
I20260929 02:51:08.390933 20217 ts_tablet_manager.cc:1939] T 73ac6adab2d94ac09e47feb30e2f1784 P 442c9269865748c9a3a5d47b46fc931e: Deleting consensus metadata
I20260929 02:51:08.393018 19699 catalog_manager.cc:4841] TS 442c9269865748c9a3a5d47b46fc931e (127.0.97.1:37243): tablet 73ac6adab2d94ac09e47feb30e2f1784 (table test-table [id=a827b36bb6f14aa7a0608b15ff6235a7]) successfully deleted
I20260929 02:51:08.394604 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.365333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57300 (local address 127.0.97.1:37243)
0929 02:51:08.365609 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.365618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.377178 (+ 11560us) server_negotiation.cc:407] Connection header received
0929 02:51:08.377381 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.377390 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.377509 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.377636 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:08.379206 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.380290 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.381711 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.382322 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.382478 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.387130 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:08.387146 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:08.387164 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:08.387210 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:08.390904 (+ 3694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.391757 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.391766 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.391769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.391931 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.392444 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.392454 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.392457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.393046 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:08.393373 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.393948 (+ 575us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.394185 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:51:08.395323 19899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:08.497787 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 19776
I20260929 02:51:08.516680 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 19910
Sep 29 02:51:08 dist-test-slave-6tjh krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260929 02:51:08.536262 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.517447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36504 (local address 127.0.97.2:44655)
0929 02:51:08.517664 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.517670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.517692 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:51:08.517914 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.517920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.517986 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.518161 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:08.519554 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.520332 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.522459 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.522926 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.523063 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.528872 (+ 5809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:08.528886 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:08.528902 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:08.528952 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:08.532505 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.533651 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.533663 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.533668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.533836 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.534319 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.534328 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.534333 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.534915 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:08.535227 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.535551 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.535802 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:51:08.537250 20033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:08.631526 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 19910
I20260929 02:51:08.648392 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20044
Sep 29 02:51:08 dist-test-slave-6tjh krb5kdc[19670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260929 02:51:08.664873 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:08.649346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44296 (local address 127.0.97.3:42691)
0929 02:51:08.649587 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:08.649594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:08.649617 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:51:08.649815 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:08.649820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:08.649897 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:08.650007 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:08.651347 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.652102 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.653534 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:08.653947 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:08.654085 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:08.658597 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:08.658614 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:08.658630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:08.658664 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:08.661811 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.662639 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.662648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.662651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.662802 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:08.663212 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:08.663218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:08.663221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:08.663593 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:08.663920 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:08.664259 (+ 339us) server_negotiation.cc:299] Negotiation successful
0929 02:51:08.664505 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:51:08.665639 20167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:08.760182 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20044
I20260929 02:51:08.776903 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 19683
I20260929 02:51:08.778147 19745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:08.891934 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 19683
2026-09-29T02:51:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4469 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T02:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:08Z Disabled control of system clock
I20260929 02:51:08.970263 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45615
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:51:09.317073 20226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:09.317390 20226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:09.317471 20226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:09.324802 20226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:09.324921 20226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:09.324961 20226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:09.325026 20226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:09.325078 20226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:09.334362 20226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45615
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:09.335668 20226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:09.337476 20226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:09.350962 20232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:09.350972 20234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:09.351188 20231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:09.351912 20226 server_base.cc:1034] running on GCE node
I20260929 02:51:09.352841 20226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:09.354383 20226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:09.355613 20226 hybrid_clock.cc:648] HybridClock initialized: now 1790650269355557 us; error 59 us; skew 500 ppm
I20260929 02:51:09.356056 20226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:09.358611 20226 webserver.cc:460] Webserver started at http://127.0.97.62:44967/ using document root <none> and password file <none>
I20260929 02:51:09.359378 20226 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:09.359476 20226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:09.359810 20226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:09.363071 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "f917d69fa0dc4cc5a5556d20efa03745"
format_stamp: "Formatted at 2026-09-29 02:51:09 on dist-test-slave-6tjh"
server_key: "13f81676a290aa54e1cd5e89858e1235"
server_key_iv: "6a47071114d3f7d47edaed3d584ff81a"
server_key_version: "encryptionkey@0"
I20260929 02:51:09.364084 20226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "f917d69fa0dc4cc5a5556d20efa03745"
format_stamp: "Formatted at 2026-09-29 02:51:09 on dist-test-slave-6tjh"
server_key: "13f81676a290aa54e1cd5e89858e1235"
server_key_iv: "6a47071114d3f7d47edaed3d584ff81a"
server_key_version: "encryptionkey@0"
I20260929 02:51:09.370285 20226 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 02:51:09.374413 20240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:09.376159 20226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:51:09.376381 20226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "f917d69fa0dc4cc5a5556d20efa03745"
format_stamp: "Formatted at 2026-09-29 02:51:09 on dist-test-slave-6tjh"
server_key: "13f81676a290aa54e1cd5e89858e1235"
server_key_iv: "6a47071114d3f7d47edaed3d584ff81a"
server_key_version: "encryptionkey@0"
I20260929 02:51:09.376659 20226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:09.390146 20226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:09.391031 20226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:09.391340 20226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:09.410774 20226 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45615
I20260929 02:51:09.410794 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45615 every 8 connection(s)
I20260929 02:51:09.412163 20226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:09.415978 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20226
I20260929 02:51:09.416011 20292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:09.416182 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:09.416864 388 external_mini_cluster.cc:1428] Setting key 39d23c5c88ba807ecbe774a3afa4381f
I20260929 02:51:09.430778 20295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:09.418864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42324 (local address 127.0.97.62:45615)
0929 02:51:09.420000 (+ 1136us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:09.420018 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:09.420061 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:51:09.421713 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:09.421737 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:09.422268 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:09.422701 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:09.423308 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:09.424690 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:09.426077 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:09.426782 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:09.426997 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:09.428314 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:09.428322 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:09.428338 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:09.429002 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:09.429084 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:09.429110 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:09.429249 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:09.429458 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 02:51:09.429807 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":397,"threads_started":1}
I20260929 02:51:09.430773 20292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745: Bootstrap starting.
I20260929 02:51:09.434568 20292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:09.435758 20292 log.cc:824] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:09.439224 20292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745: No bootstrap required, opened a new log
I20260929 02:51:09.448051 20292 raft_consensus.cc:348] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f917d69fa0dc4cc5a5556d20efa03745" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45615 } }
I20260929 02:51:09.448521 20292 raft_consensus.cc:374] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:09.448693 20292 raft_consensus.cc:729] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f917d69fa0dc4cc5a5556d20efa03745, State: Initialized, Role: FOLLOWER
I20260929 02:51:09.449424 20292 consensus_queue.cc:260] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f917d69fa0dc4cc5a5556d20efa03745" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45615 } }
I20260929 02:51:09.449730 20292 raft_consensus.cc:388] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:09.449887 20292 raft_consensus.cc:482] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:09.450083 20292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:09.453161 20292 raft_consensus.cc:504] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f917d69fa0dc4cc5a5556d20efa03745" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45615 } }
I20260929 02:51:09.453716 20292 leader_election.cc:302] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f917d69fa0dc4cc5a5556d20efa03745; no voters:
I20260929 02:51:09.454315 20292 leader_election.cc:288] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:09.454514 20297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:09.455242 20297 raft_consensus.cc:686] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [term 1 LEADER]: Becoming Leader. State: Replica: f917d69fa0dc4cc5a5556d20efa03745, State: Running, Role: LEADER
I20260929 02:51:09.455755 20297 consensus_queue.cc:237] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f917d69fa0dc4cc5a5556d20efa03745" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45615 } }
I20260929 02:51:09.456670 20292 sys_catalog.cc:564] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:09.458680 20299 sys_catalog.cc:455] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f917d69fa0dc4cc5a5556d20efa03745. Latest consensus state: current_term: 1 leader_uuid: "f917d69fa0dc4cc5a5556d20efa03745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f917d69fa0dc4cc5a5556d20efa03745" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45615 } } }
I20260929 02:51:09.459378 20298 sys_catalog.cc:455] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f917d69fa0dc4cc5a5556d20efa03745" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f917d69fa0dc4cc5a5556d20efa03745" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45615 } } }
I20260929 02:51:09.459668 20299 sys_catalog.cc:458] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:09.459977 20298 sys_catalog.cc:458] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:09.460887 20304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:09.468178 20304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:09.477662 20304 catalog_manager.cc:1347] Generated new cluster ID: f3ace83ddd674627a8a91c0cdab703fe
I20260929 02:51:09.477797 20304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:09.493695 20304 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:09.495064 20304 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:09.503322 20304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745: Generated new TSK 0
I20260929 02:51:09.504598 20304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:09.520133 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45615
--builtin_ntp_servers=127.0.97.20:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 02:51:09.875041 20316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:09.875377 20316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:09.875550 20316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:09.883317 20316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:09.883472 20316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:09.883657 20316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260929 02:51:09.893411 20316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:09.894891 20316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:09.896795 20316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:09.911643 20324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:09.911613 20321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:09.911616 20322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:09.912418 20316 server_base.cc:1034] running on GCE node
I20260929 02:51:09.913160 20316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:09.914268 20316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:09.915524 20316 hybrid_clock.cc:648] HybridClock initialized: now 1790650269915452 us; error 72 us; skew 500 ppm
I20260929 02:51:09.915904 20316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:09.923920 20316 webserver.cc:460] Webserver started at http://127.0.97.1:36371/ using document root <none> and password file <none>
I20260929 02:51:09.924863 20316 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:09.925015 20316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:09.925359 20316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:09.928429 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data/instance:
uuid: "60ba98789c2f4920af0d8ed9fa848c04"
format_stamp: "Formatted at 2026-09-29 02:51:09 on dist-test-slave-6tjh"
server_key: "36f6fd8f73b3cb090f37af352148f2fc"
server_key_iv: "ae6dbe82f04b4041f73b4a2750dd7213"
server_key_version: "encryptionkey@0"
I20260929 02:51:09.929360 20316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal/instance:
uuid: "60ba98789c2f4920af0d8ed9fa848c04"
format_stamp: "Formatted at 2026-09-29 02:51:09 on dist-test-slave-6tjh"
server_key: "36f6fd8f73b3cb090f37af352148f2fc"
server_key_iv: "ae6dbe82f04b4041f73b4a2750dd7213"
server_key_version: "encryptionkey@0"
I20260929 02:51:09.935740 20316 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:51:09.939616 20330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:09.941263 20316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:09.941473 20316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal
uuid: "60ba98789c2f4920af0d8ed9fa848c04"
format_stamp: "Formatted at 2026-09-29 02:51:09 on dist-test-slave-6tjh"
server_key: "36f6fd8f73b3cb090f37af352148f2fc"
server_key_iv: "ae6dbe82f04b4041f73b4a2750dd7213"
server_key_version: "encryptionkey@0"
I20260929 02:51:09.941725 20316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:09.963932 20316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:09.964944 20316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:09.965270 20316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:09.966359 20316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:09.967936 20316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:09.968026 20316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:09.968110 20316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:09.968168 20316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:10.003095 20316 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46447
I20260929 02:51:10.003149 20442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46447 every 8 connection(s)
I20260929 02:51:10.004534 20316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/data/info.pb
I20260929 02:51:10.006402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20316
I20260929 02:51:10.006590 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal/instance
I20260929 02:51:10.007117 388 external_mini_cluster.cc:1428] Setting key 1cdcd7a55999e123251d851f0b62d8d6
I20260929 02:51:10.010142 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45615
--builtin_ntp_servers=127.0.97.20:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:51:10.019352 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.007595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35177 (local address 127.0.97.62:45615)
0929 02:51:10.008066 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:10.008070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:10.009691 (+ 1621us) server_negotiation.cc:407] Connection header received
0929 02:51:10.011610 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:10.011618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:10.011725 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:10.011849 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:10.013021 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.014178 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.015883 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.016672 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.016814 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:10.018502 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:10.018508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:10.018511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:10.018532 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:10.018574 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:10.018587 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:10.018669 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:10.018976 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 02:51:10.019073 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260929 02:51:10.020892 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.008328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45615 (local address 127.0.97.1:35177)
0929 02:51:10.009501 (+ 1173us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:10.009548 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:10.011316 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:10.012050 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:10.012061 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:10.012157 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:10.012836 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:10.012855 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.014354 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.014358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:10.015715 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:10.015729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.016944 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.016951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:10.017202 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:10.018062 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:51:10.018068 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:10.018132 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:51:10.018256 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:10.018710 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:10.018726 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:10.018737 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:51:10.018967 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 02:51:10.019427 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":431,"threads_started":1}
I20260929 02:51:10.022650 20443 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45615
I20260929 02:51:10.023094 20443 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:10.023898 20443 heartbeater.cc:511] Master 127.0.97.62:45615 requested a full tablet report, sending...
I20260929 02:51:10.026351 20257 ts_manager.cc:194] Registered new tserver with Master: 60ba98789c2f4920af0d8ed9fa848c04 (127.0.97.1:46447)
I20260929 02:51:10.028595 20257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35177
W20260929 02:51:10.351182 20447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:10.351466 20447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:10.351559 20447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:10.359213 20447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:10.359334 20447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:10.359445 20447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260929 02:51:10.369614 20447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:10.370927 20447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:10.372802 20447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:10.386895 20453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:10.386907 20452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:10.386907 20455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:10.387759 20447 server_base.cc:1034] running on GCE node
I20260929 02:51:10.388352 20447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:10.389432 20447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:10.390643 20447 hybrid_clock.cc:648] HybridClock initialized: now 1790650270390585 us; error 69 us; skew 500 ppm
I20260929 02:51:10.390929 20447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:10.393353 20447 webserver.cc:460] Webserver started at http://127.0.97.2:42815/ using document root <none> and password file <none>
I20260929 02:51:10.394025 20447 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:10.394112 20447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:10.394374 20447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:10.397337 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data/instance:
uuid: "9b77ff85f2514521a0a7409db88524e1"
format_stamp: "Formatted at 2026-09-29 02:51:10 on dist-test-slave-6tjh"
server_key: "76b3e3b26f900ab008929bd88c7dd1ec"
server_key_iv: "2fe2f23ba520f485d41f1ddccf521576"
server_key_version: "encryptionkey@0"
I20260929 02:51:10.398180 20447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal/instance:
uuid: "9b77ff85f2514521a0a7409db88524e1"
format_stamp: "Formatted at 2026-09-29 02:51:10 on dist-test-slave-6tjh"
server_key: "76b3e3b26f900ab008929bd88c7dd1ec"
server_key_iv: "2fe2f23ba520f485d41f1ddccf521576"
server_key_version: "encryptionkey@0"
I20260929 02:51:10.403661 20447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 02:51:10.407157 20461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:10.408479 20447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:51:10.408689 20447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal
uuid: "9b77ff85f2514521a0a7409db88524e1"
format_stamp: "Formatted at 2026-09-29 02:51:10 on dist-test-slave-6tjh"
server_key: "76b3e3b26f900ab008929bd88c7dd1ec"
server_key_iv: "2fe2f23ba520f485d41f1ddccf521576"
server_key_version: "encryptionkey@0"
I20260929 02:51:10.408883 20447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:10.428469 20447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:10.429455 20447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:10.429730 20447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:10.430660 20447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:10.432135 20447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:10.432219 20447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:10.432291 20447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:10.432345 20447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:10.466528 20447 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37423
I20260929 02:51:10.466630 20573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37423 every 8 connection(s)
I20260929 02:51:10.467931 20447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/data/info.pb
I20260929 02:51:10.469893 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20447
I20260929 02:51:10.470098 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal/instance
I20260929 02:51:10.470674 388 external_mini_cluster.cc:1428] Setting key 5c99c99845ba209a22b8b1f2a657fbc6
I20260929 02:51:10.473660 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45615
--builtin_ntp_servers=127.0.97.20:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 02:51:10.482370 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.471216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49831 (local address 127.0.97.62:45615)
0929 02:51:10.471423 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:10.471432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:10.473082 (+ 1650us) server_negotiation.cc:407] Connection header received
0929 02:51:10.474938 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:10.474946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:10.475039 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:10.475150 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:10.476232 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.477392 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.479055 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.479784 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.479917 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:10.481509 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:10.481516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:10.481521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:10.481551 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:10.481582 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:10.481600 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:10.481673 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:10.481938 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 02:51:10.482047 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:51:10.483767 20576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.471823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45615 (local address 127.0.97.2:49831)
0929 02:51:10.472924 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:10.472966 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:10.474657 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:10.475353 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:10.475362 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:10.475454 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:10.476051 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:10.476069 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.477549 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.477553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:10.478911 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:10.478923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.479991 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.479995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:10.480281 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:10.481105 (+ 824us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:51:10.481107 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:10.481170 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:51:10.481291 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:10.481686 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:10.481693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:10.481699 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 02:51:10.481908 (+ 209us) client_negotiation.cc:240] Negotiation successful
0929 02:51:10.482370 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":350,"threads_started":1}
I20260929 02:51:10.485327 20574 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45615
I20260929 02:51:10.485682 20574 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:10.486474 20574 heartbeater.cc:511] Master 127.0.97.62:45615 requested a full tablet report, sending...
I20260929 02:51:10.487942 20257 ts_manager.cc:194] Registered new tserver with Master: 9b77ff85f2514521a0a7409db88524e1 (127.0.97.2:37423)
I20260929 02:51:10.488829 20257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:49831
W20260929 02:51:10.810654 20577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:10.811036 20577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:10.811193 20577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:10.818970 20577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:10.819156 20577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:10.819337 20577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260929 02:51:10.829384 20577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45615
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:10.830821 20577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:10.832692 20577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:10.846091 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:10.846165 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:10.846401 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:10.847246 20577 server_base.cc:1034] running on GCE node
I20260929 02:51:10.847924 20577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:10.848976 20577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:10.850238 20577 hybrid_clock.cc:648] HybridClock initialized: now 1790650270850192 us; error 39 us; skew 500 ppm
I20260929 02:51:10.850597 20577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:51:10.853129 20577 webserver.cc:460] Webserver started at http://127.0.97.3:41329/ using document root <none> and password file <none>
I20260929 02:51:10.853839 20577 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:10.853982 20577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:10.854310 20577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:10.857270 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data/instance:
uuid: "7ca17ad497ab4078bb2a1523b4c23803"
format_stamp: "Formatted at 2026-09-29 02:51:10 on dist-test-slave-6tjh"
server_key: "c8c5d451e1e61154b4f3857a5f0964d4"
server_key_iv: "c73cfe3c06b57d10d2373077cf4959fb"
server_key_version: "encryptionkey@0"
I20260929 02:51:10.858304 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal/instance:
uuid: "7ca17ad497ab4078bb2a1523b4c23803"
format_stamp: "Formatted at 2026-09-29 02:51:10 on dist-test-slave-6tjh"
server_key: "c8c5d451e1e61154b4f3857a5f0964d4"
server_key_iv: "c73cfe3c06b57d10d2373077cf4959fb"
server_key_version: "encryptionkey@0"
I20260929 02:51:10.864027 20577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:51:10.868012 20591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:10.869652 20577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:10.869887 20577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal
uuid: "7ca17ad497ab4078bb2a1523b4c23803"
format_stamp: "Formatted at 2026-09-29 02:51:10 on dist-test-slave-6tjh"
server_key: "c8c5d451e1e61154b4f3857a5f0964d4"
server_key_iv: "c73cfe3c06b57d10d2373077cf4959fb"
server_key_version: "encryptionkey@0"
I20260929 02:51:10.870155 20577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:10.890275 20577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:10.891188 20577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:10.891495 20577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:10.892521 20577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 02:51:10.894161 20577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:51:10.894285 20577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:10.894433 20577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 02:51:10.894522 20577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:10.928625 20577 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:44779
I20260929 02:51:10.928738 20703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:44779 every 8 connection(s)
I20260929 02:51:10.930095 20577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/data/info.pb
I20260929 02:51:10.932330 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20577
I20260929 02:51:10.932523 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal/instance
I20260929 02:51:10.933120 388 external_mini_cluster.cc:1428] Setting key e2effe7bcbcc3b7e9ed9af5075234efe
I20260929 02:51:10.944453 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.933017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:39231 (local address 127.0.97.62:45615)
0929 02:51:10.933215 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:10.933223 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:10.935176 (+ 1953us) server_negotiation.cc:407] Connection header received
0929 02:51:10.937475 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:10.937481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:10.937580 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:10.937693 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:10.938787 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.939541 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.941454 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.941882 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.942036 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:10.943570 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:10.943575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:10.943578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:10.943600 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:10.943633 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:10.943651 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:10.943751 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:10.944005 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 02:51:10.944121 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:51:10.945643 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.933753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45615 (local address 127.0.97.3:39231)
0929 02:51:10.935026 (+ 1273us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:10.935077 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:10.937189 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:10.937933 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:10.937941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:10.938045 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:10.938621 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:10.938638 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.939692 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.939697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:10.941276 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:10.941292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.942153 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.942161 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:10.942411 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:10.943206 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 02:51:10.943209 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:10.943253 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 02:51:10.943347 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:10.943751 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:10.943765 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:10.943776 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 02:51:10.943994 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 02:51:10.944465 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":429,"threads_started":1}
I20260929 02:51:10.947232 20704 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45615
I20260929 02:51:10.947590 20704 heartbeater.cc:464] Registering TS with master...
I20260929 02:51:10.948621 20704 heartbeater.cc:511] Master 127.0.97.62:45615 requested a full tablet report, sending...
I20260929 02:51:10.950080 20257 ts_manager.cc:194] Registered new tserver with Master: 7ca17ad497ab4078bb2a1523b4c23803 (127.0.97.3:44779)
I20260929 02:51:10.950867 20257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:39231
I20260929 02:51:10.961520 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 02:51:10.971944 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.965884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.0.97.62:45615)
0929 02:51:10.966092 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:10.966097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:10.966602 (+ 505us) server_negotiation.cc:407] Connection header received
0929 02:51:10.966754 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:10.966758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:10.966834 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:10.966926 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:10.967400 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.968101 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.969535 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:10.969963 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:10.970071 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:10.971208 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:10.971213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:10.971216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:10.971236 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:10.971270 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:10.971282 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:10.971384 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:10.971621 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 02:51:10.971707 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:51:10.976214 20257 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 02:51:10.983481 20257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:51:11.012764 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.997492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37423 (local address 127.0.0.1:49352)
0929 02:51:10.998808 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:10.998860 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:10.999095 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:11.003601 (+ 4506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:11.003613 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:11.003672 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:11.004118 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.004137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.006579 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.006586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.009339 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.009357 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.011587 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.011597 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.011851 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.011930 (+ 79us) client_negotiation.cc:769] Sending connection context
0929 02:51:11.012055 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 02:51:11.012314 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":5406,"threads_started":1}
I20260929 02:51:11.014034 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.997693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49352 (local address 127.0.97.2:37423)
0929 02:51:11.002801 (+ 5108us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.002812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.002850 (+ 38us) server_negotiation.cc:407] Connection header received
0929 02:51:11.002964 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.002973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.003273 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.004369 (+ 1096us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:11.004657 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.006367 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.009650 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.011312 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.013000 (+ 1688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.013279 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.013365 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.013569 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4907,"thread_start_us":4803,"threads_started":1}
I20260929 02:51:11.015450 20716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.997783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44779 (local address 127.0.0.1:54754)
0929 02:51:11.003255 (+ 5472us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:11.003278 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:11.003404 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:11.005073 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:11.005082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:11.005122 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:11.005602 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.005618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.008338 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.008347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.011322 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.011343 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.014493 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.014503 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.014726 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.014766 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 02:51:11.014866 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 02:51:11.015034 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5294,"thread_start_us":1032,"threads_started":1}
I20260929 02:51:11.015868 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:10.998498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.0.97.3:44779)
0929 02:51:11.003853 (+ 5355us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.003866 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.003912 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:51:11.004034 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.004046 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.004378 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.004722 (+ 344us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:11.005891 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.008135 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.011610 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.013884 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.014033 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.015189 (+ 1156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.015272 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.015459 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5162,"thread_start_us":3589,"threads_started":1}
I20260929 02:51:11.018395 20509 tablet_service.cc:1467] Processing CreateTablet for tablet c9b1d040652a472b9aafeb26d23b47dd (DEFAULT_TABLE table=test-table [id=9331bf50684c434980c3a198441a9a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:11.019812 20509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b1d040652a472b9aafeb26d23b47dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:11.019799 20639 tablet_service.cc:1467] Processing CreateTablet for tablet c9b1d040652a472b9aafeb26d23b47dd (DEFAULT_TABLE table=test-table [id=9331bf50684c434980c3a198441a9a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:11.021529 20639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b1d040652a472b9aafeb26d23b47dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:11.023953 20718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.007013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46447 (local address 127.0.0.1:33702)
0929 02:51:11.013403 (+ 6390us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:11.013438 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:11.013551 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:11.015250 (+ 1699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:11.015256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:11.015286 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:11.015532 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.015550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.018702 (+ 3152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.018711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.020850 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.020886 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.023061 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.023070 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.023305 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.023353 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 02:51:11.023431 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 02:51:11.023612 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6280,"thread_start_us":6782,"threads_started":1}
I20260929 02:51:11.025280 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.003646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33702 (local address 127.0.97.1:46447)
0929 02:51:11.014574 (+ 10928us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.014585 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.014619 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:51:11.014723 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.014732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.015011 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.015873 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:11.016869 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.018508 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.021148 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.022830 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.024244 (+ 1414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.024605 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.024690 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.024887 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10714,"thread_start_us":10558,"threads_started":1}
I20260929 02:51:11.029599 20378 tablet_service.cc:1467] Processing CreateTablet for tablet c9b1d040652a472b9aafeb26d23b47dd (DEFAULT_TABLE table=test-table [id=9331bf50684c434980c3a198441a9a18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 02:51:11.031330 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b1d040652a472b9aafeb26d23b47dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:11.032159 20443 heartbeater.cc:503] Master 127.0.97.62:45615 was elected leader, sending a full tablet report...
I20260929 02:51:11.038547 20722 tablet_bootstrap.cc:492] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Bootstrap starting.
I20260929 02:51:11.042310 20723 tablet_bootstrap.cc:492] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Bootstrap starting.
I20260929 02:51:11.043516 20722 tablet_bootstrap.cc:654] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:11.045164 20722 log.cc:824] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:11.047186 20723 tablet_bootstrap.cc:654] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:11.048944 20723 log.cc:824] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:11.049007 20725 tablet_bootstrap.cc:492] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Bootstrap starting.
I20260929 02:51:11.050292 20722 tablet_bootstrap.cc:492] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: No bootstrap required, opened a new log
I20260929 02:51:11.050688 20722 ts_tablet_manager.cc:1397] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 02:51:11.052217 20725 tablet_bootstrap.cc:654] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:11.052785 20723 tablet_bootstrap.cc:492] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: No bootstrap required, opened a new log
I20260929 02:51:11.053120 20723 ts_tablet_manager.cc:1397] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260929 02:51:11.053459 20725 log.cc:824] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:11.056071 20725 tablet_bootstrap.cc:492] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: No bootstrap required, opened a new log
I20260929 02:51:11.056386 20725 ts_tablet_manager.cc:1397] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 02:51:11.063663 20722 raft_consensus.cc:348] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.064285 20722 raft_consensus.cc:374] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:11.064405 20722 raft_consensus.cc:729] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ca17ad497ab4078bb2a1523b4c23803, State: Initialized, Role: FOLLOWER
I20260929 02:51:11.064308 20725 raft_consensus.cc:348] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.064785 20725 raft_consensus.cc:374] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:11.064913 20725 raft_consensus.cc:729] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ba98789c2f4920af0d8ed9fa848c04, State: Initialized, Role: FOLLOWER
I20260929 02:51:11.065356 20722 consensus_queue.cc:260] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.065508 20725 consensus_queue.cc:260] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.066016 20723 raft_consensus.cc:348] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.066676 20723 raft_consensus.cc:374] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:11.066804 20723 raft_consensus.cc:729] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b77ff85f2514521a0a7409db88524e1, State: Initialized, Role: FOLLOWER
I20260929 02:51:11.067374 20725 ts_tablet_manager.cc:1428] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Time spent starting tablet: real 0.011s user 0.008s sys 0.003s
I20260929 02:51:11.067838 20723 consensus_queue.cc:260] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.068652 20704 heartbeater.cc:503] Master 127.0.97.62:45615 was elected leader, sending a full tablet report...
I20260929 02:51:11.069449 20722 ts_tablet_manager.cc:1428] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Time spent starting tablet: real 0.019s user 0.020s sys 0.000s
I20260929 02:51:11.071157 20574 heartbeater.cc:503] Master 127.0.97.62:45615 was elected leader, sending a full tablet report...
I20260929 02:51:11.071790 20723 ts_tablet_manager.cc:1428] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Time spent starting tablet: real 0.019s user 0.015s sys 0.001s
I20260929 02:51:11.074815 20729 raft_consensus.cc:482] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:51:11.075148 20729 raft_consensus.cc:504] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.077605 20729 leader_election.cc:288] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b77ff85f2514521a0a7409db88524e1 (127.0.97.2:37423), 60ba98789c2f4920af0d8ed9fa848c04 (127.0.97.1:46447)
I20260929 02:51:11.081523 20730 raft_consensus.cc:482] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:51:11.081852 20730 raft_consensus.cc:504] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.084847 20706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.077808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37423 (local address 127.0.97.3:48671)
0929 02:51:11.078019 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:11.078038 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:11.078132 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:11.078959 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:11.078966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:11.079001 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:11.079236 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.079250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.080574 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.080581 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.082654 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.082668 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.083786 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.083792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.084111 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.084160 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 02:51:11.084277 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 02:51:11.084444 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":70}
I20260929 02:51:11.085083 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.077998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48671 (local address 127.0.97.2:37423)
0929 02:51:11.078600 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.078605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.078625 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:51:11.078686 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.078690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.078763 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.078895 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:11.079430 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.080418 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.082818 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.083626 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.083772 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.083866 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.084667 (+ 801us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.084804 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473}
I20260929 02:51:11.087467 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.079392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36487 (local address 127.0.97.1:46447)
0929 02:51:11.079576 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.079581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.079598 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:51:11.079662 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.079666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.079739 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.079826 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:11.080423 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.081573 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.084410 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.086155 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.086273 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.086373 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.086942 (+ 569us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.087103 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 02:51:11.087417 20730 leader_election.cc:288] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ca17ad497ab4078bb2a1523b4c23803 (127.0.97.3:44779), 60ba98789c2f4920af0d8ed9fa848c04 (127.0.97.1:46447)
I20260929 02:51:11.088534 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.078300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46447 (local address 127.0.97.3:36487)
0929 02:51:11.078850 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:11.078865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:11.078961 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:11.079871 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:11.079876 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:11.079904 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:11.080253 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.080264 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.081925 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.081933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.084137 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.084154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.086537 (+ 2383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.086543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.086721 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.086754 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 02:51:11.087783 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0929 02:51:11.087998 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":42,"thread_start_us":870,"threads_started":1}
I20260929 02:51:11.089761 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b1d040652a472b9aafeb26d23b47dd" candidate_uuid: "7ca17ad497ab4078bb2a1523b4c23803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba98789c2f4920af0d8ed9fa848c04" is_pre_election: true
I20260929 02:51:11.090358 20398 raft_consensus.cc:2454] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ca17ad497ab4078bb2a1523b4c23803 in term 0.
I20260929 02:51:11.091310 20592 leader_election.cc:302] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba98789c2f4920af0d8ed9fa848c04, 7ca17ad497ab4078bb2a1523b4c23803; no voters:
I20260929 02:51:11.091874 20729 raft_consensus.cc:2781] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:51:11.092151 20729 raft_consensus.cc:482] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:51:11.092296 20729 raft_consensus.cc:3037] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:11.093802 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b1d040652a472b9aafeb26d23b47dd" candidate_uuid: "7ca17ad497ab4078bb2a1523b4c23803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b77ff85f2514521a0a7409db88524e1" is_pre_election: true
I20260929 02:51:11.094367 20529 raft_consensus.cc:2454] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ca17ad497ab4078bb2a1523b4c23803 in term 0.
I20260929 02:51:11.097014 20729 raft_consensus.cc:504] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.097754 20729 leader_election.cc:288] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [CANDIDATE]: Term 1 election: Requested vote from peers 9b77ff85f2514521a0a7409db88524e1 (127.0.97.2:37423), 60ba98789c2f4920af0d8ed9fa848c04 (127.0.97.1:46447)
I20260929 02:51:11.098227 20529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b1d040652a472b9aafeb26d23b47dd" candidate_uuid: "7ca17ad497ab4078bb2a1523b4c23803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b77ff85f2514521a0a7409db88524e1"
I20260929 02:51:11.098333 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b1d040652a472b9aafeb26d23b47dd" candidate_uuid: "7ca17ad497ab4078bb2a1523b4c23803" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba98789c2f4920af0d8ed9fa848c04"
I20260929 02:51:11.098451 20529 raft_consensus.cc:3037] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:11.098589 20398 raft_consensus.cc:3037] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:11.100574 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.084397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34183 (local address 127.0.97.3:44779)
0929 02:51:11.084615 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.084621 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.093252 (+ 8631us) server_negotiation.cc:407] Connection header received
0929 02:51:11.093563 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.093569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.093654 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.093729 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:11.094738 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.095724 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.098072 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.099322 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.099424 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.099537 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.100131 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.100280 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:51:11.100631 20732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.084187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:44779 (local address 127.0.97.2:34183)
0929 02:51:11.093138 (+ 8951us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:11.093157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:11.093373 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:11.093941 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:11.093948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:11.093981 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:11.094489 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.094506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.095870 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.095874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.097830 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.097846 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.099729 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.099736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.099933 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.099968 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 02:51:11.100080 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 02:51:11.100238 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8772,"thread_start_us":8592,"threads_started":1}
I20260929 02:51:11.101423 20398 raft_consensus.cc:2454] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ca17ad497ab4078bb2a1523b4c23803 in term 1.
I20260929 02:51:11.101625 20529 raft_consensus.cc:2454] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ca17ad497ab4078bb2a1523b4c23803 in term 1.
I20260929 02:51:11.101979 20592 leader_election.cc:302] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ba98789c2f4920af0d8ed9fa848c04, 7ca17ad497ab4078bb2a1523b4c23803; no voters:
I20260929 02:51:11.102344 20729 raft_consensus.cc:2781] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:11.103047 20729 raft_consensus.cc:686] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 1 LEADER]: Becoming Leader. State: Replica: 7ca17ad497ab4078bb2a1523b4c23803, State: Running, Role: LEADER
I20260929 02:51:11.102880 20659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b1d040652a472b9aafeb26d23b47dd" candidate_uuid: "9b77ff85f2514521a0a7409db88524e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ca17ad497ab4078bb2a1523b4c23803" is_pre_election: true
I20260929 02:51:11.103785 20729 consensus_queue.cc:237] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } }
I20260929 02:51:11.108927 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.086973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46455 (local address 127.0.97.1:46447)
0929 02:51:11.087729 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.087737 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.097544 (+ 9807us) server_negotiation.cc:407] Connection header received
0929 02:51:11.098279 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.098287 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.098396 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.098507 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 02:51:11.102783 (+ 4276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.103912 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.105838 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.107023 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.107182 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.107307 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.108532 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.108701 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":9765,"threads_started":1}
I20260929 02:51:11.108986 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.086953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46447 (local address 127.0.97.2:46455)
0929 02:51:11.097402 (+ 10449us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:11.097422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:11.097563 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:11.102159 (+ 4596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:11.102164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:11.102202 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 02:51:11.102576 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.102590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.104052 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.104057 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.105626 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:11.105640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.108152 (+ 2512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.108157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:11.108324 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.108366 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 02:51:11.108465 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 02:51:11.108649 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10298,"thread_start_us":10914,"threads_started":1}
I20260929 02:51:11.109251 20255 catalog_manager.cc:5495] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ca17ad497ab4078bb2a1523b4c23803 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "7ca17ad497ab4078bb2a1523b4c23803" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ca17ad497ab4078bb2a1523b4c23803" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 44779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 } health_report { overall_health: UNKNOWN } } }
I20260929 02:51:11.111280 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b1d040652a472b9aafeb26d23b47dd" candidate_uuid: "9b77ff85f2514521a0a7409db88524e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60ba98789c2f4920af0d8ed9fa848c04" is_pre_election: true
I20260929 02:51:11.111665 20398 raft_consensus.cc:2379] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b77ff85f2514521a0a7409db88524e1 in current term 1: Already voted for candidate 7ca17ad497ab4078bb2a1523b4c23803 in this term.
I20260929 02:51:11.112603 20462 leader_election.cc:302] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b77ff85f2514521a0a7409db88524e1; no voters: 60ba98789c2f4920af0d8ed9fa848c04, 7ca17ad497ab4078bb2a1523b4c23803
I20260929 02:51:11.113097 20730 raft_consensus.cc:2726] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:51:11.146600 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.140684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54766 (local address 127.0.97.3:44779)
0929 02:51:11.140857 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.140862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.140924 (+ 62us) server_negotiation.cc:407] Connection header received
0929 02:51:11.141156 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.141160 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.141269 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.141419 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:11.141858 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.142700 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.144192 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.144745 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.144858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.145368 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:11.145914 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:11.146079 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.146191 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.146266 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:51:11.155923 20398 raft_consensus.cc:1261] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 1 FOLLOWER]: Refusing update from remote peer 7ca17ad497ab4078bb2a1523b4c23803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:11.155922 20529 raft_consensus.cc:1261] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 1 FOLLOWER]: Refusing update from remote peer 7ca17ad497ab4078bb2a1523b4c23803: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:11.156929 20735 consensus_queue.cc:1035] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b77ff85f2514521a0a7409db88524e1" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:11.157367 20729 consensus_queue.cc:1035] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60ba98789c2f4920af0d8ed9fa848c04" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:11.190270 20256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:11.190701 20256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 02:51:11.194748 20256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f917d69fa0dc4cc5a5556d20efa03745: Sending DeleteTablet for 3 replicas of tablet c9b1d040652a472b9aafeb26d23b47dd
I20260929 02:51:11.196139 20378 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b1d040652a472b9aafeb26d23b47dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:11 UTC) from {username='slave'} at 127.0.0.1:33702
I20260929 02:51:11.196120 20639 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b1d040652a472b9aafeb26d23b47dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:11 UTC) from {username='slave'} at 127.0.0.1:54754
I20260929 02:51:11.196588 20509 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b1d040652a472b9aafeb26d23b47dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:51:11 UTC) from {username='slave'} at 127.0.0.1:49352
I20260929 02:51:11.198257 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20316
I20260929 02:51:11.208719 20748 tablet_replica.cc:330] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: stopping tablet replica
I20260929 02:51:11.208961 20750 tablet_replica.cc:330] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: stopping tablet replica
I20260929 02:51:11.209699 20750 raft_consensus.cc:2229] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:11.210383 20750 raft_consensus.cc:2258] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:11.210472 20748 raft_consensus.cc:2229] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:51:11.211426 20749 tablet_replica.cc:330] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: stopping tablet replica
I20260929 02:51:11.211906 20748 raft_consensus.cc:2258] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:11.212044 20749 raft_consensus.cc:2229] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:51:11.212625 20749 raft_consensus.cc:2258] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:51:11.213253 20750 ts_tablet_manager.cc:1905] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:11.214365 20748 ts_tablet_manager.cc:1905] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:11.214788 20749 ts_tablet_manager.cc:1905] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:51:11.221807 20749 ts_tablet_manager.cc:1918] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:11.221892 20748 ts_tablet_manager.cc:1918] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:11.222098 20749 log.cc:1196] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-0/wal/wals/c9b1d040652a472b9aafeb26d23b47dd
I20260929 02:51:11.222144 20748 log.cc:1196] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-2/wal/wals/c9b1d040652a472b9aafeb26d23b47dd
I20260929 02:51:11.222747 20749 ts_tablet_manager.cc:1939] T c9b1d040652a472b9aafeb26d23b47dd P 60ba98789c2f4920af0d8ed9fa848c04: Deleting consensus metadata
I20260929 02:51:11.222879 20748 ts_tablet_manager.cc:1939] T c9b1d040652a472b9aafeb26d23b47dd P 7ca17ad497ab4078bb2a1523b4c23803: Deleting consensus metadata
I20260929 02:51:11.224761 20241 catalog_manager.cc:4841] TS 60ba98789c2f4920af0d8ed9fa848c04 (127.0.97.1:46447): tablet c9b1d040652a472b9aafeb26d23b47dd (table test-table [id=9331bf50684c434980c3a198441a9a18]) successfully deleted
I20260929 02:51:11.224933 20244 catalog_manager.cc:4841] TS 7ca17ad497ab4078bb2a1523b4c23803 (127.0.97.3:44779): tablet c9b1d040652a472b9aafeb26d23b47dd (table test-table [id=9331bf50684c434980c3a198441a9a18]) successfully deleted
I20260929 02:51:11.226613 20750 ts_tablet_manager.cc:1918] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 02:51:11.226799 20750 log.cc:1196] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Deleting WAL directory at /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790649949495923-388-0/minicluster-data/ts-1/wal/wals/c9b1d040652a472b9aafeb26d23b47dd
I20260929 02:51:11.227320 20750 ts_tablet_manager.cc:1939] T c9b1d040652a472b9aafeb26d23b47dd P 9b77ff85f2514521a0a7409db88524e1: Deleting consensus metadata
I20260929 02:51:11.229218 20241 catalog_manager.cc:4841] TS 9b77ff85f2514521a0a7409db88524e1 (127.0.97.2:37423): tablet c9b1d040652a472b9aafeb26d23b47dd (table test-table [id=9331bf50684c434980c3a198441a9a18]) successfully deleted
I20260929 02:51:11.231184 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.199587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33718 (local address 127.0.97.1:46447)
0929 02:51:11.199770 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.199777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.222275 (+ 22498us) server_negotiation.cc:407] Connection header received
0929 02:51:11.222858 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.222865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.222980 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.223085 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:11.223849 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.225214 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.227090 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.227833 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.227987 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.229297 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:11.229308 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:11.229328 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:11.230211 (+ 883us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:11.230283 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:11.230312 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:11.230444 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.230663 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.230793 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 02:51:11.231935 20438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:11.331976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20316
I20260929 02:51:11.350915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20447
I20260929 02:51:11.358927 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.351910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49366 (local address 127.0.97.2:37423)
0929 02:51:11.352137 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.352145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.352167 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:51:11.352332 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.352344 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.352441 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.352600 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:11.353107 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.353883 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.355229 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.355694 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.355847 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.357293 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:11.357303 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:11.357319 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:11.358071 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:11.358128 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:11.358150 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:11.358296 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.358446 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.358573 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:51:11.359771 20569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:11.489152 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20447
I20260929 02:51:11.507625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20577
I20260929 02:51:11.515395 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:11.508551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.0.97.3:44779)
0929 02:51:11.508753 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:11.508760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:11.508836 (+ 76us) server_negotiation.cc:407] Connection header received
0929 02:51:11.509060 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:11.509065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:11.509190 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:11.509334 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:11.509741 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.510488 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.511776 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:11.512259 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:11.512408 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:11.513846 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:11.513854 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:11.513873 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:11.514606 (+ 733us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:11.514681 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:11.514708 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:11.514827 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:11.514970 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 02:51:11.515076 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:51:11.516178 20699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:11.640271 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20577
I20260929 02:51:11.659648 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20226
I20260929 02:51:11.660840 20287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:11.780018 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20226
2026-09-29T02:51:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2891 ms)
[----------] 6 tests from EncryptionPolicyTest (22457 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:11 dist-test-slave-6tjh krb5kdc[20758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:11 dist-test-slave-6tjh krb5kdc[20758](info): set up 2 sockets
Sep 29 02:51:11 dist-test-slave-6tjh krb5kdc[20758](info): commencing operation
krb5kdc: starting...
W20260929 02:51:12.965538 388 mini_kdc.cc:121] Time spent starting KDC: real 1.137s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:13 dist-test-slave-6tjh krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:13.235602 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:13.583529 20771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:13.583978 20771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:13.584161 20771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:13.592149 20771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:13.592346 20771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:13.592455 20771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:13.592593 20771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:13.592664 20771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:13.603155 20771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41085
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:13.604586 20771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:13.605967 20771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:13.618381 20776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:13.618448 20777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:13.618419 20779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:13.618799 20771 server_base.cc:1034] running on GCE node
I20260929 02:51:13.619951 20771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:13.621634 20771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:13.622898 20771 hybrid_clock.cc:648] HybridClock initialized: now 1790650273622838 us; error 65 us; skew 500 ppm
I20260929 02:51:13.623312 20771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:13 dist-test-slave-6tjh krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:13.626688 20771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:13.628265 20771 webserver.cc:460] Webserver started at http://127.0.0.1:45101/ using document root <none> and password file <none>
I20260929 02:51:13.629012 20771 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:13.629125 20771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:13.629380 20771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:13.632408 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "0405522024444f6c9871f19337079eb2"
format_stamp: "Formatted at 2026-09-29 02:51:13 on dist-test-slave-6tjh"
server_key: "0ccdd328f4cb9d53f1dc7359e1ef8a22"
server_key_iv: "c30f8ad47ba3431170342f4d651e9eb5"
server_key_version: "encryptionkey@0"
I20260929 02:51:13.633203 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "0405522024444f6c9871f19337079eb2"
format_stamp: "Formatted at 2026-09-29 02:51:13 on dist-test-slave-6tjh"
server_key: "0ccdd328f4cb9d53f1dc7359e1ef8a22"
server_key_iv: "c30f8ad47ba3431170342f4d651e9eb5"
server_key_version: "encryptionkey@0"
I20260929 02:51:13.638566 20771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:51:13.642447 20786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:13.643937 20771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:51:13.644182 20771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "0405522024444f6c9871f19337079eb2"
format_stamp: "Formatted at 2026-09-29 02:51:13 on dist-test-slave-6tjh"
server_key: "0ccdd328f4cb9d53f1dc7359e1ef8a22"
server_key_iv: "c30f8ad47ba3431170342f4d651e9eb5"
server_key_version: "encryptionkey@0"
I20260929 02:51:13.644412 20771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:13.669127 20771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:13.675002 20771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:13.675364 20771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:13.695286 20771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41085
I20260929 02:51:13.695317 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41085 every 8 connection(s)
I20260929 02:51:13.696794 20771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:13.700748 20838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:13.702277 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20771
I20260929 02:51:13.702515 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:13.703328 388 external_mini_cluster.cc:1428] Setting key 26e7f902dee1b779dbf65973cbc5a008
I20260929 02:51:13.716113 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2: Bootstrap starting.
Sep 29 02:51:13 dist-test-slave-6tjh krb5kdc[20758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:51:13.720310 20838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:13.721621 20838 log.cc:824] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:13.725791 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2: No bootstrap required, opened a new log
I20260929 02:51:13.731681 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:13.705423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41973 (local address 127.0.0.1:41085)
0929 02:51:13.706310 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:13.706324 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:13.706369 (+ 45us) server_negotiation.cc:407] Connection header received
0929 02:51:13.708038 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:13.708061 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:13.708491 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:13.708912 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:13.710948 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:13.712309 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:13.714281 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:13.714985 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:13.715127 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:13.720206 (+ 5079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:13.720214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:13.720230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:13.720275 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:13.724587 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:13.725997 (+ 1410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:13.726037 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:13.726049 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:13.726298 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:13.726996 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:13.727003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:13.727007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:13.727904 (+ 897us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:13.729399 (+ 1495us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:13.729775 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 02:51:13.730164 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":413,"threads_started":1}
I20260929 02:51:13.736335 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:13.736589 20838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0405522024444f6c9871f19337079eb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260929 02:51:13.737088 20838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:13.737215 20838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0405522024444f6c9871f19337079eb2, State: Initialized, Role: FOLLOWER
I20260929 02:51:13.737902 20838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0405522024444f6c9871f19337079eb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260929 02:51:13.738183 20838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:13.738343 20838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:13.738569 20838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:13.741667 20838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0405522024444f6c9871f19337079eb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260929 02:51:13.742449 20838 leader_election.cc:302] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0405522024444f6c9871f19337079eb2; no voters:
I20260929 02:51:13.743167 20838 leader_election.cc:288] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:13.743305 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:13.745141 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0405522024444f6c9871f19337079eb2, State: Running, Role: LEADER
I20260929 02:51:13.745699 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0405522024444f6c9871f19337079eb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } }
I20260929 02:51:13.745784 20838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:13.750772 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0405522024444f6c9871f19337079eb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0405522024444f6c9871f19337079eb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } } }
I20260929 02:51:13.751278 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:13.753185 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0405522024444f6c9871f19337079eb2. Latest consensus state: current_term: 1 leader_uuid: "0405522024444f6c9871f19337079eb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0405522024444f6c9871f19337079eb2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41085 } } }
I20260929 02:51:13.753568 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:13.761260 20856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:13.763571 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:13.741451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44635 (local address 127.0.0.1:41085)
0929 02:51:13.741624 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:13.741629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:13.742263 (+ 634us) server_negotiation.cc:407] Connection header received
0929 02:51:13.742422 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:13.742443 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:13.742538 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:13.742663 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:13.744869 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:13.746177 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:13.747842 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:13.749033 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:13.749191 (+ 158us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:13.753608 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:13.753617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:13.753625 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:13.753685 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:13.758703 (+ 5018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:13.759845 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:13.759852 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:13.759856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:13.760018 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:13.760593 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:13.760600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:13.760604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:13.761175 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:13.762519 (+ 1344us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:13.762755 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 02:51:13.763005 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 02:51:13.768904 20856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:13.779336 20856 catalog_manager.cc:1347] Generated new cluster ID: f0af046331da47a495239e343693accf
I20260929 02:51:13.779490 20856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:13.808342 20856 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:13.810546 20856 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:13.825659 20856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0405522024444f6c9871f19337079eb2: Generated new TSK 0
I20260929 02:51:13.826927 20856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:13.839609 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20771
I20260929 02:51:13.840884 20833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:13.937469 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20771
W20260929 02:51:13.955683 20756 connection.cc:535] client connection to 127.0.0.1:41085 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41085): Connection reset by peer (error 104)
2026-09-29T02:51:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2148 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:14 dist-test-slave-6tjh krb5kdc[20871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:14 dist-test-slave-6tjh krb5kdc[20871](info): set up 2 sockets
Sep 29 02:51:14 dist-test-slave-6tjh krb5kdc[20871](info): commencing operation
krb5kdc: starting...
W20260929 02:51:15.065279 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:15 dist-test-slave-6tjh krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:15.336318 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:15.687340 20884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:15.687674 20884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:15.687851 20884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:15.695694 20884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:15.695835 20884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:15.695930 20884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:15.696040 20884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:15.696137 20884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:15.706533 20884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:15.707935 20884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:15.709393 20884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:15.721544 20889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:15.721602 20890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:15.721602 20892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:15.723029 20884 server_base.cc:1034] running on GCE node
I20260929 02:51:15.724162 20884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:15.726308 20884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:15.727677 20884 hybrid_clock.cc:648] HybridClock initialized: now 1790650275727598 us; error 63 us; skew 500 ppm
I20260929 02:51:15.728176 20884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:15 dist-test-slave-6tjh krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650275, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:15.735889 20884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:15.737702 20884 webserver.cc:460] Webserver started at http://127.0.0.1:33347/ using document root <none> and password file <none>
I20260929 02:51:15.738516 20884 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:15.738678 20884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:15.739007 20884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:15.742214 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "c5552ce94bb1498ba6dba934e07f63b6"
format_stamp: "Formatted at 2026-09-29 02:51:15 on dist-test-slave-6tjh"
server_key: "97c4419f33d7f1835db2c36ea8a7da8a"
server_key_iv: "0d9f9c91ab0667a210f21f9975bf9dd5"
server_key_version: "encryptionkey@0"
I20260929 02:51:15.742992 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5552ce94bb1498ba6dba934e07f63b6"
format_stamp: "Formatted at 2026-09-29 02:51:15 on dist-test-slave-6tjh"
server_key: "97c4419f33d7f1835db2c36ea8a7da8a"
server_key_iv: "0d9f9c91ab0667a210f21f9975bf9dd5"
server_key_version: "encryptionkey@0"
I20260929 02:51:15.748199 20884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 02:51:15.752027 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:15.753615 20884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 02:51:15.753821 20884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "c5552ce94bb1498ba6dba934e07f63b6"
format_stamp: "Formatted at 2026-09-29 02:51:15 on dist-test-slave-6tjh"
server_key: "97c4419f33d7f1835db2c36ea8a7da8a"
server_key_iv: "0d9f9c91ab0667a210f21f9975bf9dd5"
server_key_version: "encryptionkey@0"
I20260929 02:51:15.754063 20884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:15.773641 20884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:15.778133 20884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:15.778443 20884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:15.798209 20884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45565
I20260929 02:51:15.798233 20950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45565 every 8 connection(s)
I20260929 02:51:15.799748 20884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:15.803222 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20884
I20260929 02:51:15.803421 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:15.803987 388 external_mini_cluster.cc:1428] Setting key bdee6bb519fddba97798e944828df0a0
I20260929 02:51:15.804147 20951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:15.819943 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6: Bootstrap starting.
I20260929 02:51:15.820279 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:15.806177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46489 (local address 127.0.0.1:45565)
0929 02:51:15.807226 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:15.807244 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:15.807631 (+ 387us) server_negotiation.cc:407] Connection header received
0929 02:51:15.809496 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:15.809539 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:15.810186 (+ 647us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:15.810707 (+ 521us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:15.811301 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:15.813359 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:15.814825 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:15.815936 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:15.816135 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:15.817420 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:15.817432 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:15.817452 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:15.818198 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:15.818302 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:15.818333 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:15.818498 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:15.818687 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 02:51:15.818913 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":454,"threads_started":1}
I20260929 02:51:15.823606 20951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:15.824491 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:15.824916 20951 log.cc:824] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:15.828476 20951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6: No bootstrap required, opened a new log
I20260929 02:51:15.837500 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:15.829617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48891 (local address 127.0.0.1:45565)
0929 02:51:15.829853 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:15.829859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:15.830213 (+ 354us) server_negotiation.cc:407] Connection header received
0929 02:51:15.830445 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:15.830449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:15.830511 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:15.830600 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:15.831321 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:15.832110 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:15.834495 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:15.834994 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:15.835110 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:15.836524 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:15.836528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:15.836534 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:15.836609 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:15.836659 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:15.836675 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:15.836771 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:15.837074 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 02:51:15.837161 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260929 02:51:15.840478 20951 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5552ce94bb1498ba6dba934e07f63b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260929 02:51:15.841040 20951 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:15.841171 20951 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5552ce94bb1498ba6dba934e07f63b6, State: Initialized, Role: FOLLOWER
I20260929 02:51:15.841838 20951 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5552ce94bb1498ba6dba934e07f63b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260929 02:51:15.842149 20951 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:15.842329 20951 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:15.842545 20951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:15.845655 20951 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5552ce94bb1498ba6dba934e07f63b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260929 02:51:15.846331 20951 leader_election.cc:302] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5552ce94bb1498ba6dba934e07f63b6; no voters:
I20260929 02:51:15.846951 20951 leader_election.cc:288] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:15.847059 20961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:15.847755 20961 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [term 1 LEADER]: Becoming Leader. State: Replica: c5552ce94bb1498ba6dba934e07f63b6, State: Running, Role: LEADER
I20260929 02:51:15.848399 20961 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5552ce94bb1498ba6dba934e07f63b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } }
I20260929 02:51:15.849627 20951 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:15.851473 20962 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5552ce94bb1498ba6dba934e07f63b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5552ce94bb1498ba6dba934e07f63b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } } }
I20260929 02:51:15.851986 20962 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:15.851523 20963 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5552ce94bb1498ba6dba934e07f63b6. Latest consensus state: current_term: 1 leader_uuid: "c5552ce94bb1498ba6dba934e07f63b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5552ce94bb1498ba6dba934e07f63b6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45565 } } }
I20260929 02:51:15.852630 20963 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:15.853550 20967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:15.862397 20967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:15.872978 20967 catalog_manager.cc:1347] Generated new cluster ID: ac6216903a074f05bf216e8b1671656d
I20260929 02:51:15.873162 20967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:15.904888 20967 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:15.906389 20967 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:15.916785 20967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5552ce94bb1498ba6dba934e07f63b6: Generated new TSK 0
I20260929 02:51:15.917891 20967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:15.929724 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20884
W20260929 02:51:15.931675 20946 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46489
W20260929 02:51:15.932492 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20884: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:15.932746 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20884
W20260929 02:51:15.949805 20869 connection.cc:535] client connection to 127.0.0.1:45565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45565): Connection reset by peer (error 104)
2026-09-29T02:51:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:16 dist-test-slave-6tjh krb5kdc[20982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:16 dist-test-slave-6tjh krb5kdc[20982](info): set up 2 sockets
Sep 29 02:51:16 dist-test-slave-6tjh krb5kdc[20982](info): commencing operation
krb5kdc: starting...
W20260929 02:51:17.044773 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:17 dist-test-slave-6tjh krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:17.307884 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45449
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:17.647361 20995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:17.647732 20995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:17.647907 20995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:17.655263 20995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:17.655439 20995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:17.655555 20995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:17.655659 20995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:17.655746 20995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:17.665781 20995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45449
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:17.667166 20995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:17.668514 20995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:17.680678 21000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:17.680752 21001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:17.680640 21003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:17.681603 20995 server_base.cc:1034] running on GCE node
I20260929 02:51:17.682443 20995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:17.683924 20995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:17.685171 20995 hybrid_clock.cc:648] HybridClock initialized: now 1790650277685103 us; error 44 us; skew 500 ppm
I20260929 02:51:17.685524 20995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:17 dist-test-slave-6tjh krb5kdc[20982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650277, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:17.689008 20995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:17.690656 20995 webserver.cc:460] Webserver started at http://127.0.0.1:44109/ using document root <none> and password file <none>
I20260929 02:51:17.691370 20995 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:17.691519 20995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:17.691839 20995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:17.694957 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "515bb6b68e6c4775bb7347869ddd064e"
format_stamp: "Formatted at 2026-09-29 02:51:17 on dist-test-slave-6tjh"
server_key: "1200e9c6911505c02b016a9ec054d50a"
server_key_iv: "df0009b6ddf92bf317b8e435015b4855"
server_key_version: "encryptionkey@0"
I20260929 02:51:17.695752 20995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "515bb6b68e6c4775bb7347869ddd064e"
format_stamp: "Formatted at 2026-09-29 02:51:17 on dist-test-slave-6tjh"
server_key: "1200e9c6911505c02b016a9ec054d50a"
server_key_iv: "df0009b6ddf92bf317b8e435015b4855"
server_key_version: "encryptionkey@0"
I20260929 02:51:17.701186 20995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:51:17.704859 21010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:17.706192 20995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:51:17.706386 20995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "515bb6b68e6c4775bb7347869ddd064e"
format_stamp: "Formatted at 2026-09-29 02:51:17 on dist-test-slave-6tjh"
server_key: "1200e9c6911505c02b016a9ec054d50a"
server_key_iv: "df0009b6ddf92bf317b8e435015b4855"
server_key_version: "encryptionkey@0"
I20260929 02:51:17.706597 20995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:17.712687 20995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:17.716987 20995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:17.717307 20995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:17.738744 20995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45449
I20260929 02:51:17.738768 21061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45449 every 8 connection(s)
I20260929 02:51:17.740268 20995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:17.744364 21062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:17.744715 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 20995
I20260929 02:51:17.744897 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:17.745358 388 external_mini_cluster.cc:1428] Setting key 382ac3ecbb3f2fea012b40b4ea7eff20
I20260929 02:51:17.753471 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:17.747200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51925 (local address 127.0.0.1:45449)
0929 02:51:17.748046 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:17.748058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:17.748104 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:51:17.749820 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:17.749844 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:17.750213 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:17.750647 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:17.751729 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:17.751736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:17.751749 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:17.751763 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:17.751829 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:17.751851 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:17.752102 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:17.752253 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 02:51:17.752449 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":376,"threads_started":1}
I20260929 02:51:17.757556 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:17.758760 21062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e: Bootstrap starting.
I20260929 02:51:17.762151 21062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:17.763264 21062 log.cc:824] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:17.764464 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:17.761949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51949 (local address 127.0.0.1:45449)
0929 02:51:17.762167 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:17.762174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:17.762418 (+ 244us) server_negotiation.cc:407] Connection header received
0929 02:51:17.762618 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:17.762624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:17.762684 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:17.762788 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:17.763776 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:17.763782 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:17.763786 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:17.763811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:17.763851 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:17.763866 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:17.763994 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:17.764112 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 02:51:17.764190 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:51:17.766561 21062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e: No bootstrap required, opened a new log
I20260929 02:51:17.774240 21062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515bb6b68e6c4775bb7347869ddd064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45449 } }
I20260929 02:51:17.774639 21062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:17.774722 21062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515bb6b68e6c4775bb7347869ddd064e, State: Initialized, Role: FOLLOWER
I20260929 02:51:17.775315 21062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515bb6b68e6c4775bb7347869ddd064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45449 } }
I20260929 02:51:17.775548 21062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:17.775676 21062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:17.775802 21062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:17.778414 21062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515bb6b68e6c4775bb7347869ddd064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45449 } }
I20260929 02:51:17.778877 21062 leader_election.cc:302] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 515bb6b68e6c4775bb7347869ddd064e; no voters:
I20260929 02:51:17.779469 21062 leader_election.cc:288] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:17.779671 21072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:17.780453 21072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [term 1 LEADER]: Becoming Leader. State: Replica: 515bb6b68e6c4775bb7347869ddd064e, State: Running, Role: LEADER
I20260929 02:51:17.781049 21072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515bb6b68e6c4775bb7347869ddd064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45449 } }
I20260929 02:51:17.781551 21062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:17.783844 21074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 515bb6b68e6c4775bb7347869ddd064e. Latest consensus state: current_term: 1 leader_uuid: "515bb6b68e6c4775bb7347869ddd064e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515bb6b68e6c4775bb7347869ddd064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45449 } } }
I20260929 02:51:17.784297 21074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:17.784034 21073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "515bb6b68e6c4775bb7347869ddd064e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "515bb6b68e6c4775bb7347869ddd064e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45449 } } }
I20260929 02:51:17.784432 21073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:17.787534 21080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:17.793288 21080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:17.803696 21080 catalog_manager.cc:1347] Generated new cluster ID: c18803279da54e1a984e96086c23dfc4
I20260929 02:51:17.803858 21080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:17.822342 21080 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:17.822947 21080 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:17.836635 21080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 515bb6b68e6c4775bb7347869ddd064e: Generated new TSK 0
I20260929 02:51:17.837644 21080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:17.851884 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20995
W20260929 02:51:17.853288 21057 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51925
W20260929 02:51:17.854144 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 20995: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:17.854354 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 20995
2026-09-29T02:51:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1921 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:17 dist-test-slave-6tjh krb5kdc[21093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:17 dist-test-slave-6tjh krb5kdc[21093](info): set up 2 sockets
Sep 29 02:51:17 dist-test-slave-6tjh krb5kdc[21093](info): commencing operation
krb5kdc: starting...
W20260929 02:51:18.981042 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:19 dist-test-slave-6tjh krb5kdc[21093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:19.253129 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:19.601454 21106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:19.601902 21106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:19.602066 21106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:19.609704 21106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:19.609828 21106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:19.609932 21106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:19.610039 21106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:19.610118 21106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:19.620236 21106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:19.621610 21106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:19.622973 21106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:19.635444 21114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:19.635380 21112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:19.635419 21111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:19.636338 21106 server_base.cc:1034] running on GCE node
I20260929 02:51:19.637219 21106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:19.638792 21106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:19.640033 21106 hybrid_clock.cc:648] HybridClock initialized: now 1790650279640004 us; error 67 us; skew 500 ppm
I20260929 02:51:19.640430 21106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:19 dist-test-slave-6tjh krb5kdc[21093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:19.643690 21106 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:19.645380 21106 webserver.cc:460] Webserver started at http://127.0.0.1:41459/ using document root <none> and password file <none>
I20260929 02:51:19.646100 21106 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:19.646253 21106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:19.646567 21106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:19.649705 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "0138089f3a794636a3f9dcf86bd0f21f"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-6tjh"
server_key: "d0f4debab0ba12f9dc5f78e3e51d0263"
server_key_iv: "08a954830ce85390fac6b1e5f2db21fe"
server_key_version: "encryptionkey@0"
I20260929 02:51:19.650501 21106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "0138089f3a794636a3f9dcf86bd0f21f"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-6tjh"
server_key: "d0f4debab0ba12f9dc5f78e3e51d0263"
server_key_iv: "08a954830ce85390fac6b1e5f2db21fe"
server_key_version: "encryptionkey@0"
I20260929 02:51:19.655790 21106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:51:19.659305 21121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:19.660742 21106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:19.660923 21106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "0138089f3a794636a3f9dcf86bd0f21f"
format_stamp: "Formatted at 2026-09-29 02:51:19 on dist-test-slave-6tjh"
server_key: "d0f4debab0ba12f9dc5f78e3e51d0263"
server_key_iv: "08a954830ce85390fac6b1e5f2db21fe"
server_key_version: "encryptionkey@0"
I20260929 02:51:19.661121 21106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:19.683005 21106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:19.687460 21106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:19.687783 21106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:19.707034 21106 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42325
I20260929 02:51:19.707080 21172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42325 every 8 connection(s)
I20260929 02:51:19.708473 21106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:19.709199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21106
I20260929 02:51:19.709416 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:19.710007 388 external_mini_cluster.cc:1428] Setting key fadef4909a9038d3f67552c9cf372849
I20260929 02:51:19.713452 21173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:51:19 dist-test-slave-6tjh krb5kdc[21093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:51:19.728956 21173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f: Bootstrap starting.
I20260929 02:51:19.733632 21173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:19.735109 21173 log.cc:824] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:19.738956 21173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f: No bootstrap required, opened a new log
I20260929 02:51:19.739207 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:19.711924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50471 (local address 127.0.0.1:42325)
0929 02:51:19.712945 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:19.712962 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:19.713024 (+ 62us) server_negotiation.cc:407] Connection header received
0929 02:51:19.715395 (+ 2371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:19.715425 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:19.715972 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:19.716488 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:19.718086 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.720224 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.721775 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.722828 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.723026 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:19.729460 (+ 6434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:19.729474 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:19.729523 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:19.729586 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:19.733951 (+ 4365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:19.735101 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:19.735111 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:19.735121 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:19.735364 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:19.735936 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:19.735945 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:19.735948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:19.736606 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:19.736933 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:19.737368 (+ 435us) server_negotiation.cc:299] Negotiation successful
0929 02:51:19.737825 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":409,"threads_started":1}
I20260929 02:51:19.743471 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:19.747359 21173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0138089f3a794636a3f9dcf86bd0f21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260929 02:51:19.747828 21173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:19.747969 21173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0138089f3a794636a3f9dcf86bd0f21f, State: Initialized, Role: FOLLOWER
I20260929 02:51:19.748661 21173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0138089f3a794636a3f9dcf86bd0f21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260929 02:51:19.748987 21173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:19.749159 21173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:19.749364 21173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:19.752228 21173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0138089f3a794636a3f9dcf86bd0f21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260929 02:51:19.752820 21173 leader_election.cc:302] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0138089f3a794636a3f9dcf86bd0f21f; no voters:
I20260929 02:51:19.753458 21173 leader_election.cc:288] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:19.753578 21183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:19.754715 21183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [term 1 LEADER]: Becoming Leader. State: Replica: 0138089f3a794636a3f9dcf86bd0f21f, State: Running, Role: LEADER
I20260929 02:51:19.755301 21183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0138089f3a794636a3f9dcf86bd0f21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } }
I20260929 02:51:19.755925 21173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:19.757943 21185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0138089f3a794636a3f9dcf86bd0f21f. Latest consensus state: current_term: 1 leader_uuid: "0138089f3a794636a3f9dcf86bd0f21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0138089f3a794636a3f9dcf86bd0f21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } } }
I20260929 02:51:19.758332 21185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:19.758854 21184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0138089f3a794636a3f9dcf86bd0f21f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0138089f3a794636a3f9dcf86bd0f21f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42325 } } }
I20260929 02:51:19.759136 21184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:19.761761 21188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:19.768442 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:19.748013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48643 (local address 127.0.0.1:42325)
0929 02:51:19.748203 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:19.748209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:19.748726 (+ 517us) server_negotiation.cc:407] Connection header received
0929 02:51:19.748870 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:19.748876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:19.748960 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:19.749077 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:19.750601 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.751734 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.753165 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:19.753982 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:19.754913 (+ 931us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:19.757403 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:19.757412 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:19.757420 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:19.757473 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:19.762439 (+ 4966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:19.763604 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:19.763613 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:19.763619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:19.764009 (+ 390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:19.764706 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:19.764715 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:19.764721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:19.765420 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:19.765895 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:19.767402 (+ 1507us) server_negotiation.cc:299] Negotiation successful
0929 02:51:19.767741 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:51:19.771231 21188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:19.782713 21188 catalog_manager.cc:1347] Generated new cluster ID: 67a5bdaa488c46b493b58a2fb7512764
I20260929 02:51:19.782928 21188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:19.805709 21188 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:19.807080 21188 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:19.817263 21188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0138089f3a794636a3f9dcf86bd0f21f: Generated new TSK 0
I20260929 02:51:19.818126 21188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:19.827164 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21106
I20260929 02:51:19.828531 21168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:19.923344 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21106
2026-09-29T02:51:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2069 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:20 dist-test-slave-6tjh krb5kdc[21206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:20 dist-test-slave-6tjh krb5kdc[21206](info): set up 2 sockets
Sep 29 02:51:20 dist-test-slave-6tjh krb5kdc[21206](info): commencing operation
krb5kdc: starting...
W20260929 02:51:21.037009 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:21 dist-test-slave-6tjh krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:21.308629 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40971
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40971
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:21.658365 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:21.658725 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:21.658900 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:21.666302 21219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:21.666484 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:21.666603 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:21.666714 21219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:21.666792 21219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:21.677151 21219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:21.678577 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:21.680014 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:21.693184 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:21.693212 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:21.693288 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:21.694518 21219 server_base.cc:1034] running on GCE node
I20260929 02:51:21.695483 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:21.697650 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:21.699043 21219 hybrid_clock.cc:648] HybridClock initialized: now 1790650281698938 us; error 93 us; skew 500 ppm
I20260929 02:51:21.699556 21219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:21 dist-test-slave-6tjh krb5kdc[21206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650281, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:21.709455 21219 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:21.711350 21219 webserver.cc:460] Webserver started at http://127.0.0.1:33551/ using document root <none> and password file <none>
I20260929 02:51:21.712155 21219 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:21.712316 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:21.712752 21219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:21.716307 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "68213b75114241e98dd228a217ec7fc9"
format_stamp: "Formatted at 2026-09-29 02:51:21 on dist-test-slave-6tjh"
server_key: "6558c4337254663b0f6af4f2ae9d6e7b"
server_key_iv: "779163ae2c001e3cf503f9113a987efe"
server_key_version: "encryptionkey@0"
I20260929 02:51:21.717344 21219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "68213b75114241e98dd228a217ec7fc9"
format_stamp: "Formatted at 2026-09-29 02:51:21 on dist-test-slave-6tjh"
server_key: "6558c4337254663b0f6af4f2ae9d6e7b"
server_key_iv: "779163ae2c001e3cf503f9113a987efe"
server_key_version: "encryptionkey@0"
I20260929 02:51:21.723560 21219 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:51:21.729699 21234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:21.731338 21219 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 02:51:21.731521 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "68213b75114241e98dd228a217ec7fc9"
format_stamp: "Formatted at 2026-09-29 02:51:21 on dist-test-slave-6tjh"
server_key: "6558c4337254663b0f6af4f2ae9d6e7b"
server_key_iv: "779163ae2c001e3cf503f9113a987efe"
server_key_version: "encryptionkey@0"
I20260929 02:51:21.731729 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:21.744482 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:21.749184 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:21.749432 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:21.768332 21219 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40971
I20260929 02:51:21.768361 21285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40971 every 8 connection(s)
I20260929 02:51:21.769832 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:21.773764 21286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:21.775394 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21219
I20260929 02:51:21.775638 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:21.776119 388 external_mini_cluster.cc:1428] Setting key 4f72ee19587e4c112540ded884b74451
I20260929 02:51:21.788161 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9: Bootstrap starting.
I20260929 02:51:21.790413 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:21.778035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45015 (local address 127.0.0.1:40971)
0929 02:51:21.778862 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:21.778874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:21.778920 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:51:21.780664 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:21.780692 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:21.781159 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:21.781509 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:21.782140 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:21.783422 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:21.785468 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:21.786422 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:21.786582 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:21.787892 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:21.787900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:21.787917 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:21.788604 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:21.788705 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:21.788735 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:21.788833 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:21.789092 (+ 259us) server_negotiation.cc:299] Negotiation successful
0929 02:51:21.789393 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":396,"threads_started":1}
I20260929 02:51:21.792142 21286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:21.793476 21286 log.cc:824] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:21.794946 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:21.796634 21286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9: No bootstrap required, opened a new log
I20260929 02:51:21.806107 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:21.799355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60821 (local address 127.0.0.1:40971)
0929 02:51:21.799557 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:21.799574 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:21.799934 (+ 360us) server_negotiation.cc:407] Connection header received
0929 02:51:21.800094 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:21.800101 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:21.800182 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:21.800309 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:21.800849 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:21.801992 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:21.803387 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:21.804089 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:21.804237 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:21.805393 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:21.805400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:21.805405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:21.805427 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:21.805474 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:21.805494 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:21.805594 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:21.805680 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 02:51:21.805775 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:51:21.806367 21286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68213b75114241e98dd228a217ec7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260929 02:51:21.806896 21286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:21.807029 21286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68213b75114241e98dd228a217ec7fc9, State: Initialized, Role: FOLLOWER
I20260929 02:51:21.807780 21286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68213b75114241e98dd228a217ec7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260929 02:51:21.808041 21286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:21.808172 21286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:21.808296 21286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:21.810607 21286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68213b75114241e98dd228a217ec7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260929 02:51:21.811125 21286 leader_election.cc:302] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68213b75114241e98dd228a217ec7fc9; no voters:
I20260929 02:51:21.811698 21286 leader_election.cc:288] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:21.811977 21296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:21.813166 21296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [term 1 LEADER]: Becoming Leader. State: Replica: 68213b75114241e98dd228a217ec7fc9, State: Running, Role: LEADER
I20260929 02:51:21.813928 21296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68213b75114241e98dd228a217ec7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } }
I20260929 02:51:21.814268 21286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:21.817198 21297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68213b75114241e98dd228a217ec7fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68213b75114241e98dd228a217ec7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } } }
I20260929 02:51:21.817327 21298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68213b75114241e98dd228a217ec7fc9. Latest consensus state: current_term: 1 leader_uuid: "68213b75114241e98dd228a217ec7fc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68213b75114241e98dd228a217ec7fc9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40971 } } }
I20260929 02:51:21.817596 21297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:21.817680 21298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:21.818602 21305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:21.824786 21305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:21.834878 21305 catalog_manager.cc:1347] Generated new cluster ID: 466944ede1d2484d87ecbd7d8f1f8df1
I20260929 02:51:21.835016 21305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:21.849284 21305 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:21.851233 21305 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:21.874691 21305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68213b75114241e98dd228a217ec7fc9: Generated new TSK 0
I20260929 02:51:21.875700 21305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:21.890735 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21219
W20260929 02:51:21.891857 21281 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45015
W20260929 02:51:21.892654 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21219: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:21.892807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21219
2026-09-29T02:51:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1969 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:21 dist-test-slave-6tjh krb5kdc[21317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:21 dist-test-slave-6tjh krb5kdc[21317](info): set up 2 sockets
Sep 29 02:51:21 dist-test-slave-6tjh krb5kdc[21317](info): commencing operation
krb5kdc: starting...
W20260929 02:51:23.009284 388 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:23 dist-test-slave-6tjh krb5kdc[21317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:23.284526 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:23.631776 21330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:23.632103 21330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:23.632280 21330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:23.639830 21330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:23.639963 21330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:23.640053 21330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:23.640168 21330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:23.640267 21330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:23.650640 21330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:23.652050 21330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:23.653532 21330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:23.665834 21335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:23.665920 21338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:23.665896 21336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:23.666766 21330 server_base.cc:1034] running on GCE node
I20260929 02:51:23.667544 21330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:23.669071 21330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:23.670282 21330 hybrid_clock.cc:648] HybridClock initialized: now 1790650283670222 us; error 51 us; skew 500 ppm
I20260929 02:51:23.670647 21330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:23 dist-test-slave-6tjh krb5kdc[21317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650283, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:23.673795 21330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:23.675316 21330 webserver.cc:460] Webserver started at http://127.0.0.1:42145/ using document root <none> and password file <none>
I20260929 02:51:23.676015 21330 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:23.676163 21330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:23.676483 21330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:23.679483 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "7c9004f3ba4546868ca8345c1cd7da05"
format_stamp: "Formatted at 2026-09-29 02:51:23 on dist-test-slave-6tjh"
server_key: "d283b854305a63698e616d1f48b7e99a"
server_key_iv: "1adff0ca06e75ebe43ffd2b6887c9f35"
server_key_version: "encryptionkey@0"
I20260929 02:51:23.680279 21330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "7c9004f3ba4546868ca8345c1cd7da05"
format_stamp: "Formatted at 2026-09-29 02:51:23 on dist-test-slave-6tjh"
server_key: "d283b854305a63698e616d1f48b7e99a"
server_key_iv: "1adff0ca06e75ebe43ffd2b6887c9f35"
server_key_version: "encryptionkey@0"
I20260929 02:51:23.685802 21330 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:51:23.689958 21345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:23.691536 21330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:51:23.691769 21330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "7c9004f3ba4546868ca8345c1cd7da05"
format_stamp: "Formatted at 2026-09-29 02:51:23 on dist-test-slave-6tjh"
server_key: "d283b854305a63698e616d1f48b7e99a"
server_key_iv: "1adff0ca06e75ebe43ffd2b6887c9f35"
server_key_version: "encryptionkey@0"
I20260929 02:51:23.691999 21330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:23.698807 21330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:23.703627 21330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:23.703987 21330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:23.724813 21330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33819
I20260929 02:51:23.724886 21396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33819 every 8 connection(s)
I20260929 02:51:23.727044 21330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:23.731400 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21330
I20260929 02:51:23.731587 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:23.732148 388 external_mini_cluster.cc:1428] Setting key f8a9927e1a704943a44b4735629dc3b0
I20260929 02:51:23.732218 21397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:23.740955 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:23.734736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36933 (local address 127.0.0.1:33819)
0929 02:51:23.735626 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:23.735638 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:23.735681 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:51:23.737406 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:23.737431 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:23.737802 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:23.738153 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:23.739274 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:23.739280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:23.739294 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:23.739308 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:23.739368 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:23.739390 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:23.739584 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:23.739681 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 02:51:23.739915 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":416,"threads_started":1}
I20260929 02:51:23.744808 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:23.746500 21397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05: Bootstrap starting.
I20260929 02:51:23.750926 21397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:23.752377 21397 log.cc:824] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:23.752802 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:23.750195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57951 (local address 127.0.0.1:33819)
0929 02:51:23.750416 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:23.750425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:23.750722 (+ 297us) server_negotiation.cc:407] Connection header received
0929 02:51:23.750916 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:23.750921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:23.750973 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:23.751104 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:23.752064 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:23.752068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:23.752072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:23.752087 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:23.752127 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:23.752145 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:23.752240 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:23.752393 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 02:51:23.752447 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 02:51:23.755926 21397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05: No bootstrap required, opened a new log
I20260929 02:51:23.763727 21397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9004f3ba4546868ca8345c1cd7da05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260929 02:51:23.764168 21397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:23.764290 21397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9004f3ba4546868ca8345c1cd7da05, State: Initialized, Role: FOLLOWER
I20260929 02:51:23.764920 21397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9004f3ba4546868ca8345c1cd7da05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260929 02:51:23.765184 21397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:23.765326 21397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:23.765509 21397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:23.767719 21397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9004f3ba4546868ca8345c1cd7da05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260929 02:51:23.768345 21397 leader_election.cc:302] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c9004f3ba4546868ca8345c1cd7da05; no voters:
I20260929 02:51:23.768968 21397 leader_election.cc:288] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:23.769115 21407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:23.769842 21407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [term 1 LEADER]: Becoming Leader. State: Replica: 7c9004f3ba4546868ca8345c1cd7da05, State: Running, Role: LEADER
I20260929 02:51:23.770390 21407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9004f3ba4546868ca8345c1cd7da05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260929 02:51:23.771202 21397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:23.773288 21409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c9004f3ba4546868ca8345c1cd7da05. Latest consensus state: current_term: 1 leader_uuid: "7c9004f3ba4546868ca8345c1cd7da05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9004f3ba4546868ca8345c1cd7da05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } } }
I20260929 02:51:23.773774 21409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:23.774392 21408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c9004f3ba4546868ca8345c1cd7da05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c9004f3ba4546868ca8345c1cd7da05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } } }
I20260929 02:51:23.774747 21408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:23.776095 21414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:23.780714 21414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:23.790169 21414 catalog_manager.cc:1347] Generated new cluster ID: afe7dcea6f4c45e5a56c54ae818fc05a
I20260929 02:51:23.790305 21414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:23.805151 21414 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:23.805698 21414 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:23.818080 21414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c9004f3ba4546868ca8345c1cd7da05: Generated new TSK 0
I20260929 02:51:23.819267 21414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:23.834265 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21330
W20260929 02:51:23.835428 21392 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36933
W20260929 02:51:23.836210 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21330: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:23.836380 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21330
2026-09-29T02:51:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:23 dist-test-slave-6tjh krb5kdc[21428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:23 dist-test-slave-6tjh krb5kdc[21428](info): set up 2 sockets
Sep 29 02:51:23 dist-test-slave-6tjh krb5kdc[21428](info): commencing operation
krb5kdc: starting...
W20260929 02:51:25.013548 388 mini_kdc.cc:121] Time spent starting KDC: real 1.140s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:25 dist-test-slave-6tjh krb5kdc[21428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:25.288097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:25.627588 21441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:25.627918 21441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:25.628134 21441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:25.635663 21441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:25.635793 21441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:25.635875 21441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:25.635974 21441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:25.636052 21441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:25.646143 21441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:25.647528 21441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:25.648895 21441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:25.661605 21447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:25.661664 21449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:25.661681 21446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:25.662279 21441 server_base.cc:1034] running on GCE node
I20260929 02:51:25.663203 21441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:25.664754 21441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:25.665971 21441 hybrid_clock.cc:648] HybridClock initialized: now 1790650285665928 us; error 37 us; skew 500 ppm
I20260929 02:51:25.666370 21441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:25 dist-test-slave-6tjh krb5kdc[21428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650285, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:25.669773 21441 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:25.671283 21441 webserver.cc:460] Webserver started at http://127.0.0.1:35633/ using document root <none> and password file <none>
I20260929 02:51:25.671955 21441 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:25.672056 21441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:25.672317 21441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:25.675436 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "7cdb70012c9343238865e3a2bc88f13c"
format_stamp: "Formatted at 2026-09-29 02:51:25 on dist-test-slave-6tjh"
server_key: "4f50dca3a00aac1752fc39fa20caeadb"
server_key_iv: "10ec0179f5e7551bfd348fde34bc83d9"
server_key_version: "encryptionkey@0"
I20260929 02:51:25.676187 21441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "7cdb70012c9343238865e3a2bc88f13c"
format_stamp: "Formatted at 2026-09-29 02:51:25 on dist-test-slave-6tjh"
server_key: "4f50dca3a00aac1752fc39fa20caeadb"
server_key_iv: "10ec0179f5e7551bfd348fde34bc83d9"
server_key_version: "encryptionkey@0"
I20260929 02:51:25.681902 21441 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:51:25.685762 21456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:25.687268 21441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:51:25.687455 21441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "7cdb70012c9343238865e3a2bc88f13c"
format_stamp: "Formatted at 2026-09-29 02:51:25 on dist-test-slave-6tjh"
server_key: "4f50dca3a00aac1752fc39fa20caeadb"
server_key_iv: "10ec0179f5e7551bfd348fde34bc83d9"
server_key_version: "encryptionkey@0"
I20260929 02:51:25.687654 21441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:25.705945 21441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:25.710589 21441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:25.710896 21441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:25.736184 21441 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44203
I20260929 02:51:25.736208 21507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44203 every 8 connection(s)
I20260929 02:51:25.737706 21441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:25.741647 21508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:25.744395 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21441
I20260929 02:51:25.744606 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:25.745138 388 external_mini_cluster.cc:1428] Setting key 657af6898a20863d78d613d00ae0c0f1
I20260929 02:51:25.754359 21508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c: Bootstrap starting.
I20260929 02:51:25.758375 21508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:25.759804 21508 log.cc:824] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:51:25 dist-test-slave-6tjh krb5kdc[21428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:51:25.763507 21508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c: No bootstrap required, opened a new log
I20260929 02:51:25.770905 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:25.747015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:60913 (local address 127.0.0.1:44203)
0929 02:51:25.747831 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:25.747843 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:25.747892 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:51:25.749625 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:25.749649 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:25.750088 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:25.750446 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:25.752256 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:25.753667 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:25.755453 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:25.756215 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:25.756361 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:25.761738 (+ 5377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:25.761749 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:25.761770 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:25.761817 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:25.765786 (+ 3969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:25.766672 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:25.766682 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:25.766691 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:25.766907 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:25.767352 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:25.767358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:25.767364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:25.768124 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:25.768450 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:25.769039 (+ 589us) server_negotiation.cc:299] Negotiation successful
0929 02:51:25.769405 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":404,"threads_started":1}
I20260929 02:51:25.774268 21508 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb70012c9343238865e3a2bc88f13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260929 02:51:25.774753 21508 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:25.774868 21508 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdb70012c9343238865e3a2bc88f13c, State: Initialized, Role: FOLLOWER
I20260929 02:51:25.775588 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:25.775556 21508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb70012c9343238865e3a2bc88f13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260929 02:51:25.775826 21508 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:25.775979 21508 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:25.776145 21508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:25.778395 21508 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb70012c9343238865e3a2bc88f13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260929 02:51:25.778925 21508 leader_election.cc:302] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cdb70012c9343238865e3a2bc88f13c; no voters:
I20260929 02:51:25.779583 21508 leader_election.cc:288] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:25.779731 21517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:25.781822 21517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [term 1 LEADER]: Becoming Leader. State: Replica: 7cdb70012c9343238865e3a2bc88f13c, State: Running, Role: LEADER
I20260929 02:51:25.781899 21508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:25.782394 21517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb70012c9343238865e3a2bc88f13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } }
I20260929 02:51:25.785164 21519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cdb70012c9343238865e3a2bc88f13c. Latest consensus state: current_term: 1 leader_uuid: "7cdb70012c9343238865e3a2bc88f13c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb70012c9343238865e3a2bc88f13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } } }
I20260929 02:51:25.785562 21519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:25.786087 21518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cdb70012c9343238865e3a2bc88f13c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdb70012c9343238865e3a2bc88f13c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44203 } } }
I20260929 02:51:25.786399 21518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:25.786746 21524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:25.798333 21524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:25.802398 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:25.780921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:47753 (local address 127.0.0.1:44203)
0929 02:51:25.781095 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:25.781102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:25.781666 (+ 564us) server_negotiation.cc:407] Connection header received
0929 02:51:25.781978 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:25.781985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:25.782084 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:25.782158 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:25.784663 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:25.785975 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:25.787874 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:25.788509 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:25.788628 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:25.792856 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:25.792866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:25.792873 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:25.792921 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:25.797788 (+ 4867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:25.798957 (+ 1169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:25.798963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:25.798970 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:25.799125 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:25.799753 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:25.799757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:25.799760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:25.800220 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:25.801620 (+ 1400us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:25.801814 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 02:51:25.802011 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 02:51:25.811673 21524 catalog_manager.cc:1347] Generated new cluster ID: 7aea7ce6457a4290939cca67b693e354
I20260929 02:51:25.811808 21524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:25.837808 21524 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:25.839715 21524 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:25.848394 21524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cdb70012c9343238865e3a2bc88f13c: Generated new TSK 0
I20260929 02:51:25.849361 21524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:25.861397 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21441
I20260929 02:51:25.863338 21503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:25.956925 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21441
W20260929 02:51:25.974151 21426 connection.cc:535] client connection to 127.0.0.1:44203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44203): Connection reset by peer (error 104)
2026-09-29T02:51:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2120 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:26 dist-test-slave-6tjh krb5kdc[21540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:26 dist-test-slave-6tjh krb5kdc[21540](info): set up 2 sockets
Sep 29 02:51:26 dist-test-slave-6tjh krb5kdc[21540](info): commencing operation
krb5kdc: starting...
W20260929 02:51:27.069164 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:27 dist-test-slave-6tjh krb5kdc[21540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:27.340394 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:27.678942 21553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:27.679347 21553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:27.679530 21553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:27.687161 21553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:27.687335 21553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:27.687445 21553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:27.687554 21553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:27.687630 21553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:27.698065 21553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:27.699541 21553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:27.701020 21553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:27.713897 21561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:27.713992 21559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:27.713961 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:27.714592 21553 server_base.cc:1034] running on GCE node
I20260929 02:51:27.715443 21553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:27.717136 21553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:27.718415 21553 hybrid_clock.cc:648] HybridClock initialized: now 1790650287718352 us; error 54 us; skew 500 ppm
I20260929 02:51:27.718842 21553 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:27 dist-test-slave-6tjh krb5kdc[21540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650287, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:27.730791 21553 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:27.733265 21553 webserver.cc:460] Webserver started at http://127.0.0.1:39465/ using document root <none> and password file <none>
I20260929 02:51:27.734347 21553 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:27.734520 21553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:27.734953 21553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:27.738555 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "5831529dc15d40d293edd699d9fc292d"
format_stamp: "Formatted at 2026-09-29 02:51:27 on dist-test-slave-6tjh"
server_key: "4cd58ac3e9e8531dd4a8ba342f30bb8a"
server_key_iv: "872956fe95052043fb0bf82fee380825"
server_key_version: "encryptionkey@0"
I20260929 02:51:27.739405 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "5831529dc15d40d293edd699d9fc292d"
format_stamp: "Formatted at 2026-09-29 02:51:27 on dist-test-slave-6tjh"
server_key: "4cd58ac3e9e8531dd4a8ba342f30bb8a"
server_key_iv: "872956fe95052043fb0bf82fee380825"
server_key_version: "encryptionkey@0"
I20260929 02:51:27.745172 21553 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:51:27.749011 21568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:27.750504 21553 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:51:27.750727 21553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "5831529dc15d40d293edd699d9fc292d"
format_stamp: "Formatted at 2026-09-29 02:51:27 on dist-test-slave-6tjh"
server_key: "4cd58ac3e9e8531dd4a8ba342f30bb8a"
server_key_iv: "872956fe95052043fb0bf82fee380825"
server_key_version: "encryptionkey@0"
I20260929 02:51:27.750962 21553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:27.766640 21553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:27.771617 21553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:27.771965 21553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:27.792227 21553 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37819
I20260929 02:51:27.792245 21619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37819 every 8 connection(s)
I20260929 02:51:27.793869 21553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:27.797483 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21553
I20260929 02:51:27.797698 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:27.798058 21620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:27.798234 388 external_mini_cluster.cc:1428] Setting key 66ffa0e9c3c27937fe82901e051a91a0
I20260929 02:51:27.814019 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:27.800017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:59707 (local address 127.0.0.1:37819)
0929 02:51:27.800856 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:27.800871 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:27.800926 (+ 55us) server_negotiation.cc:407] Connection header received
0929 02:51:27.803206 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:27.803246 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:27.803855 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:27.804346 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:27.804960 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:27.807005 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:27.808470 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:27.809606 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:27.809837 (+ 231us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:27.811068 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:27.811084 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:27.811105 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:27.811889 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:27.811992 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:27.812033 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:27.812188 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:27.812458 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 02:51:27.812684 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":330,"threads_started":1}
I20260929 02:51:27.814244 21620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d: Bootstrap starting.
I20260929 02:51:27.817740 21620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:27.818620 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:27.818993 21620 log.cc:824] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:27.822687 21620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d: No bootstrap required, opened a new log
I20260929 02:51:27.831220 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:27.823734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:36793 (local address 127.0.0.1:37819)
0929 02:51:27.823958 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:27.823967 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:27.824465 (+ 498us) server_negotiation.cc:407] Connection header received
0929 02:51:27.824705 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:27.824709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:27.824779 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:27.824870 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:27.825518 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:27.826316 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:27.828340 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:27.828939 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:27.829043 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:27.830382 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:27.830387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:27.830390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:27.830409 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:27.830450 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:27.830465 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:27.830554 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:27.830771 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 02:51:27.830818 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:51:27.833235 21620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5831529dc15d40d293edd699d9fc292d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37819 } }
I20260929 02:51:27.833655 21620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:27.833776 21620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5831529dc15d40d293edd699d9fc292d, State: Initialized, Role: FOLLOWER
I20260929 02:51:27.834367 21620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5831529dc15d40d293edd699d9fc292d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37819 } }
I20260929 02:51:27.834619 21620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:27.834777 21620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:27.834947 21620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:27.837393 21620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5831529dc15d40d293edd699d9fc292d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37819 } }
I20260929 02:51:27.837939 21620 leader_election.cc:302] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5831529dc15d40d293edd699d9fc292d; no voters:
I20260929 02:51:27.838567 21620 leader_election.cc:288] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:27.838761 21630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:27.839551 21630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [term 1 LEADER]: Becoming Leader. State: Replica: 5831529dc15d40d293edd699d9fc292d, State: Running, Role: LEADER
I20260929 02:51:27.840143 21630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5831529dc15d40d293edd699d9fc292d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37819 } }
I20260929 02:51:27.840925 21620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:27.843226 21632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5831529dc15d40d293edd699d9fc292d. Latest consensus state: current_term: 1 leader_uuid: "5831529dc15d40d293edd699d9fc292d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5831529dc15d40d293edd699d9fc292d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37819 } } }
I20260929 02:51:27.843215 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5831529dc15d40d293edd699d9fc292d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5831529dc15d40d293edd699d9fc292d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37819 } } }
I20260929 02:51:27.843672 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:27.843669 21632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:27.849502 21635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:27.855137 21635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:27.865912 21635 catalog_manager.cc:1347] Generated new cluster ID: baf606154453400dbdf89d48f3af87f8
I20260929 02:51:27.866047 21635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:27.891786 21635 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:27.893687 21635 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:27.903084 21635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5831529dc15d40d293edd699d9fc292d: Generated new TSK 0
I20260929 02:51:27.904084 21635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:27.911146 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21553
W20260929 02:51:27.912592 21615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:59707
W20260929 02:51:27.913502 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:27.913668 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21553
W20260929 02:51:27.930792 21538 connection.cc:535] client connection to 127.0.0.1:37819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37819): Connection reset by peer (error 104)
2026-09-29T02:51:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:28 dist-test-slave-6tjh krb5kdc[21651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:28 dist-test-slave-6tjh krb5kdc[21651](info): set up 2 sockets
Sep 29 02:51:28 dist-test-slave-6tjh krb5kdc[21651](info): commencing operation
krb5kdc: starting...
W20260929 02:51:29.021119 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:29 dist-test-slave-6tjh krb5kdc[21651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:29.305446 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:29.658484 21664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:29.658955 21664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:29.659157 21664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:29.666846 21664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:29.666987 21664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:29.667037 21664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:29.667100 21664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:29.667143 21664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:29.677546 21664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:29.678843 21664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:29.680197 21664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:29.692835 21670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:29.692831 21672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:29.692871 21669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:29.693621 21664 server_base.cc:1034] running on GCE node
I20260929 02:51:29.694489 21664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:29.695983 21664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:29.697185 21664 hybrid_clock.cc:648] HybridClock initialized: now 1790650289697139 us; error 36 us; skew 500 ppm
I20260929 02:51:29.697538 21664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:29 dist-test-slave-6tjh krb5kdc[21651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:29.700878 21664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:29.702417 21664 webserver.cc:460] Webserver started at http://127.0.0.1:34217/ using document root <none> and password file <none>
I20260929 02:51:29.703114 21664 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:29.703259 21664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:29.703564 21664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:29.706558 21664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "75dbcba8f8ef4bacbf09a56b43078165"
format_stamp: "Formatted at 2026-09-29 02:51:29 on dist-test-slave-6tjh"
server_key: "31c75259e39e286c6fd1e90031958ea0"
server_key_iv: "0517902a9ae6728da07c7a553c8ea015"
server_key_version: "encryptionkey@0"
I20260929 02:51:29.707333 21664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "75dbcba8f8ef4bacbf09a56b43078165"
format_stamp: "Formatted at 2026-09-29 02:51:29 on dist-test-slave-6tjh"
server_key: "31c75259e39e286c6fd1e90031958ea0"
server_key_iv: "0517902a9ae6728da07c7a553c8ea015"
server_key_version: "encryptionkey@0"
I20260929 02:51:29.712793 21664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:51:29.716308 21679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:29.717896 21664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:29.718099 21664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "75dbcba8f8ef4bacbf09a56b43078165"
format_stamp: "Formatted at 2026-09-29 02:51:29 on dist-test-slave-6tjh"
server_key: "31c75259e39e286c6fd1e90031958ea0"
server_key_iv: "0517902a9ae6728da07c7a553c8ea015"
server_key_version: "encryptionkey@0"
I20260929 02:51:29.718312 21664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:29.724879 21664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:29.731386 21664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:29.731693 21664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:29.751667 21664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46783
I20260929 02:51:29.751721 21730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46783 every 8 connection(s)
I20260929 02:51:29.753248 21664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:29.757192 21731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:29.762633 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21664
I20260929 02:51:29.762828 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:29.763316 388 external_mini_cluster.cc:1428] Setting key 1bed7873c9b4024645fbc32a1bbfa48a
I20260929 02:51:29.768852 21731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165: Bootstrap starting.
I20260929 02:51:29.772131 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:29.765665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:59117 (local address 127.0.0.1:46783)
0929 02:51:29.766564 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:29.766578 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:29.766627 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:51:29.768419 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:29.768444 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:29.768911 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:29.769327 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:29.770477 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:29.770483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:29.770497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:29.770510 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:29.770590 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:29.770611 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:29.770797 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:29.770932 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 02:51:29.771162 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":404,"threads_started":1}
I20260929 02:51:29.772941 21731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:29.774106 21731 log.cc:824] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:29.776091 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:29.777281 21731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165: No bootstrap required, opened a new log
I20260929 02:51:29.783600 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:29.780793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:49491 (local address 127.0.0.1:46783)
0929 02:51:29.781004 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:29.781013 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:29.781409 (+ 396us) server_negotiation.cc:407] Connection header received
0929 02:51:29.781634 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:29.781638 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:29.781699 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:29.781823 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:29.782847 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:29.782852 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:29.782857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:29.782876 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:29.782933 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:29.782949 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:29.783105 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:29.783242 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 02:51:29.783312 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:51:29.786290 21731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dbcba8f8ef4bacbf09a56b43078165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260929 02:51:29.786738 21731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:29.786854 21731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75dbcba8f8ef4bacbf09a56b43078165, State: Initialized, Role: FOLLOWER
I20260929 02:51:29.787508 21731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dbcba8f8ef4bacbf09a56b43078165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260929 02:51:29.787782 21731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:29.787930 21731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:29.788120 21731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:29.790907 21731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dbcba8f8ef4bacbf09a56b43078165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260929 02:51:29.791462 21731 leader_election.cc:302] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75dbcba8f8ef4bacbf09a56b43078165; no voters:
I20260929 02:51:29.792153 21731 leader_election.cc:288] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:29.792325 21741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:29.793116 21741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [term 1 LEADER]: Becoming Leader. State: Replica: 75dbcba8f8ef4bacbf09a56b43078165, State: Running, Role: LEADER
I20260929 02:51:29.793763 21741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dbcba8f8ef4bacbf09a56b43078165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } }
I20260929 02:51:29.794534 21731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:29.797117 21742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75dbcba8f8ef4bacbf09a56b43078165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dbcba8f8ef4bacbf09a56b43078165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } } }
I20260929 02:51:29.797083 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75dbcba8f8ef4bacbf09a56b43078165. Latest consensus state: current_term: 1 leader_uuid: "75dbcba8f8ef4bacbf09a56b43078165" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75dbcba8f8ef4bacbf09a56b43078165" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46783 } } }
I20260929 02:51:29.797737 21742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:29.797744 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:29.798975 21750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:29.805656 21750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:29.816509 21750 catalog_manager.cc:1347] Generated new cluster ID: e2a7d6d7bc8f44f9a7b407885f0e4293
I20260929 02:51:29.816725 21750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:29.834405 21750 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:29.834893 21750 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:29.843806 21750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75dbcba8f8ef4bacbf09a56b43078165: Generated new TSK 0
I20260929 02:51:29.844820 21750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:29.851545 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21664
W20260929 02:51:29.852883 21726 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:59117
W20260929 02:51:29.853760 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:29.853947 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21664
2026-09-29T02:51:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1939 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:29 dist-test-slave-6tjh krb5kdc[21762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:29 dist-test-slave-6tjh krb5kdc[21762](info): set up 2 sockets
Sep 29 02:51:29 dist-test-slave-6tjh krb5kdc[21762](info): commencing operation
krb5kdc: starting...
W20260929 02:51:30.960999 388 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.006s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:31 dist-test-slave-6tjh krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:31.228523 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38243
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:31.576368 21775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:31.576771 21775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:31.576941 21775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:31.584297 21775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:31.584455 21775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:31.584595 21775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:31.584708 21775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:31.584786 21775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:31.594679 21775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38243
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:31.596022 21775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:31.597442 21775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:31.610656 21781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:31.610636 21783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:31.610687 21780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:31.611510 21775 server_base.cc:1034] running on GCE node
I20260929 02:51:31.612341 21775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:31.613921 21775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:31.615159 21775 hybrid_clock.cc:648] HybridClock initialized: now 1790650291615118 us; error 40 us; skew 500 ppm
I20260929 02:51:31.615523 21775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:31 dist-test-slave-6tjh krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650291, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:31.618680 21775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:31.620312 21775 webserver.cc:460] Webserver started at http://127.0.0.1:38677/ using document root <none> and password file <none>
I20260929 02:51:31.621094 21775 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:31.621244 21775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:31.621560 21775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:31.624643 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "f73277f563b54052be005e3eeaed0b77"
format_stamp: "Formatted at 2026-09-29 02:51:31 on dist-test-slave-6tjh"
server_key: "2be38d16499577ed6a7c2d7b4d8c5530"
server_key_iv: "ae978718a58b4b2302f4adb25a20d51a"
server_key_version: "encryptionkey@0"
I20260929 02:51:31.625460 21775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "f73277f563b54052be005e3eeaed0b77"
format_stamp: "Formatted at 2026-09-29 02:51:31 on dist-test-slave-6tjh"
server_key: "2be38d16499577ed6a7c2d7b4d8c5530"
server_key_iv: "ae978718a58b4b2302f4adb25a20d51a"
server_key_version: "encryptionkey@0"
I20260929 02:51:31.630867 21775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:51:31.634704 21790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:31.636271 21775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:51:31.636484 21775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "f73277f563b54052be005e3eeaed0b77"
format_stamp: "Formatted at 2026-09-29 02:51:31 on dist-test-slave-6tjh"
server_key: "2be38d16499577ed6a7c2d7b4d8c5530"
server_key_iv: "ae978718a58b4b2302f4adb25a20d51a"
server_key_version: "encryptionkey@0"
I20260929 02:51:31.636746 21775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:31.671784 21775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:31.676141 21775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:31.676452 21775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:31.695385 21775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38243
I20260929 02:51:31.695487 21841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38243 every 8 connection(s)
I20260929 02:51:31.696905 21775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:31.700639 21842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:31.705936 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21775
I20260929 02:51:31.706141 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:31.706665 388 external_mini_cluster.cc:1428] Setting key 01c9a73c63bf5dc74056075167a67f1a
I20260929 02:51:31.712538 21842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77: Bootstrap starting.
I20260929 02:51:31.716310 21842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:31.717916 21842 log.cc:824] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:31.722100 21842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77: No bootstrap required, opened a new log
Sep 29 02:51:31 dist-test-slave-6tjh krb5kdc[21762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 02:51:31.733335 21842 raft_consensus.cc:348] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73277f563b54052be005e3eeaed0b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38243 } }
I20260929 02:51:31.733834 21842 raft_consensus.cc:374] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:31.733964 21842 raft_consensus.cc:729] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f73277f563b54052be005e3eeaed0b77, State: Initialized, Role: FOLLOWER
I20260929 02:51:31.734730 21842 consensus_queue.cc:260] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73277f563b54052be005e3eeaed0b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38243 } }
I20260929 02:51:31.735030 21842 raft_consensus.cc:388] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:31.735181 21842 raft_consensus.cc:482] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:31.735360 21842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:31.736685 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:31.708625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:45399 (local address 127.0.0.1:38243)
0929 02:51:31.709478 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:31.709491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:31.709540 (+ 49us) server_negotiation.cc:407] Connection header received
0929 02:51:31.711368 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:31.711392 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:31.711827 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:31.712238 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:31.713810 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:31.715400 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:31.717729 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:31.719089 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:31.720091 (+ 1002us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:31.726136 (+ 6045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:31.726148 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:31.726175 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:31.726235 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:31.730769 (+ 4534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:31.732066 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:31.732076 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:31.732087 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:31.732318 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:31.732965 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:31.732973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:31.732976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:31.733777 (+ 801us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:31.734142 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:31.734690 (+ 548us) server_negotiation.cc:299] Negotiation successful
0929 02:51:31.735089 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":393,"threads_started":1}
I20260929 02:51:31.738721 21842 raft_consensus.cc:504] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73277f563b54052be005e3eeaed0b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38243 } }
I20260929 02:51:31.739296 21842 leader_election.cc:302] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f73277f563b54052be005e3eeaed0b77; no voters:
I20260929 02:51:31.739969 21842 leader_election.cc:288] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:31.740089 21847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:31.741168 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:31.741436 21847 raft_consensus.cc:686] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [term 1 LEADER]: Becoming Leader. State: Replica: f73277f563b54052be005e3eeaed0b77, State: Running, Role: LEADER
I20260929 02:51:31.742000 21847 consensus_queue.cc:237] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73277f563b54052be005e3eeaed0b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38243 } }
I20260929 02:51:31.742236 21842 sys_catalog.cc:564] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:31.746894 21849 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f73277f563b54052be005e3eeaed0b77. Latest consensus state: current_term: 1 leader_uuid: "f73277f563b54052be005e3eeaed0b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73277f563b54052be005e3eeaed0b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38243 } } }
I20260929 02:51:31.747233 21849 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:31.747512 21848 sys_catalog.cc:455] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f73277f563b54052be005e3eeaed0b77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f73277f563b54052be005e3eeaed0b77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38243 } } }
I20260929 02:51:31.747896 21848 sys_catalog.cc:458] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:31.754530 21857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:31.765379 21857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:31.770696 21845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:31.748405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:50719 (local address 127.0.0.1:38243)
0929 02:51:31.748653 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:31.748663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:31.753634 (+ 4971us) server_negotiation.cc:407] Connection header received
0929 02:51:31.753915 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:31.753924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:31.754042 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:31.754162 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:31.755716 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:31.756516 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:31.758291 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:31.758898 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:31.759038 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:31.762077 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:31.762087 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:31.762094 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:31.762140 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:31.766164 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:31.767314 (+ 1150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:31.767320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:31.767323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:31.767474 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:31.768102 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:31.768109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:31.768114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:31.768536 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:31.768832 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:31.770130 (+ 1298us) server_negotiation.cc:299] Negotiation successful
0929 02:51:31.770315 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:51:31.777709 21857 catalog_manager.cc:1347] Generated new cluster ID: 251de2de3c364f6fb3450b851ada9adb
I20260929 02:51:31.777832 21857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:31.796149 21857 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:31.798161 21857 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:31.816104 21857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f73277f563b54052be005e3eeaed0b77: Generated new TSK 0
I20260929 02:51:31.817306 21857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:31.822834 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21775
I20260929 02:51:31.824044 21837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:31.918901 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21775
W20260929 02:51:31.936424 21760 connection.cc:535] client connection to 127.0.0.1:38243 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38243): Connection reset by peer (error 104)
2026-09-29T02:51:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2065 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:32 dist-test-slave-6tjh krb5kdc[21874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:32 dist-test-slave-6tjh krb5kdc[21874](info): set up 2 sockets
Sep 29 02:51:32 dist-test-slave-6tjh krb5kdc[21874](info): commencing operation
krb5kdc: starting...
W20260929 02:51:33.029472 388 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:33 dist-test-slave-6tjh krb5kdc[21874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:33.317598 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:33.655035 21887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:33.655344 21887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:33.655679 21887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:33.664219 21887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:33.664367 21887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:33.664505 21887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:33.664619 21887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:33.664736 21887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:33.675179 21887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44963
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:33.676661 21887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:33.678195 21887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:33.691931 21895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:33.691965 21892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:33.691921 21893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:33.692898 21887 server_base.cc:1034] running on GCE node
I20260929 02:51:33.693739 21887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:33.695286 21887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:33.696524 21887 hybrid_clock.cc:648] HybridClock initialized: now 1790650293696485 us; error 57 us; skew 500 ppm
I20260929 02:51:33.696954 21887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:33 dist-test-slave-6tjh krb5kdc[21874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:33.704144 21887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:33.705917 21887 webserver.cc:460] Webserver started at http://127.0.0.1:39537/ using document root <none> and password file <none>
I20260929 02:51:33.706713 21887 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:33.706869 21887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:33.707186 21887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:33.710453 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3"
format_stamp: "Formatted at 2026-09-29 02:51:33 on dist-test-slave-6tjh"
server_key: "af38920dbd3def15d74e54a316fe503b"
server_key_iv: "429eb101a8eff01b6820c525f43b54da"
server_key_version: "encryptionkey@0"
I20260929 02:51:33.711290 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3"
format_stamp: "Formatted at 2026-09-29 02:51:33 on dist-test-slave-6tjh"
server_key: "af38920dbd3def15d74e54a316fe503b"
server_key_iv: "429eb101a8eff01b6820c525f43b54da"
server_key_version: "encryptionkey@0"
I20260929 02:51:33.716943 21887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 02:51:33.720731 21902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:33.722193 21887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:51:33.722414 21887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3"
format_stamp: "Formatted at 2026-09-29 02:51:33 on dist-test-slave-6tjh"
server_key: "af38920dbd3def15d74e54a316fe503b"
server_key_iv: "429eb101a8eff01b6820c525f43b54da"
server_key_version: "encryptionkey@0"
I20260929 02:51:33.722656 21887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:33.753427 21887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:33.757830 21887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:33.758142 21887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:33.777252 21887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44963
I20260929 02:51:33.777284 21953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44963 every 8 connection(s)
I20260929 02:51:33.778764 21887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:33.782858 21954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:33.783695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21887
I20260929 02:51:33.783908 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:33.788932 388 external_mini_cluster.cc:1428] Setting key 8512b8279717c53ffd647e893cd47a11
I20260929 02:51:33.794262 21954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3: Bootstrap starting.
I20260929 02:51:33.798147 21954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:33.799381 21954 log.cc:824] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:33.802707 21954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3: No bootstrap required, opened a new log
I20260929 02:51:33.803201 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:33.790894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:40159 (local address 127.0.0.1:44963)
0929 02:51:33.791768 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:33.791781 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:33.791832 (+ 51us) server_negotiation.cc:407] Connection header received
0929 02:51:33.793710 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:33.793737 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:33.794184 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:33.794586 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:33.795199 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:33.796510 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:33.798453 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:33.799150 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:33.799312 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:33.800763 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:33.800776 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:33.800798 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:33.801384 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:33.801462 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:33.801487 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:33.801606 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:33.801946 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 02:51:33.802145 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":399,"threads_started":1}
I20260929 02:51:33.807482 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:33.811311 21954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260929 02:51:33.811821 21954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:33.811954 21954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417fa3a53e9e4fa98bfe1dd3693d22e3, State: Initialized, Role: FOLLOWER
I20260929 02:51:33.812775 21954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260929 02:51:33.813086 21954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:33.813241 21954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:33.813419 21954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:33.816638 21954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260929 02:51:33.817189 21954 leader_election.cc:302] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 417fa3a53e9e4fa98bfe1dd3693d22e3; no voters:
I20260929 02:51:33.817876 21954 leader_election.cc:288] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:33.817996 21964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:33.819610 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:33.811876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:55501 (local address 127.0.0.1:44963)
0929 02:51:33.812100 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:33.812107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:33.812475 (+ 368us) server_negotiation.cc:407] Connection header received
0929 02:51:33.812703 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:33.812707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:33.812775 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:33.812856 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:33.813542 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:33.814338 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:33.816256 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:33.816760 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:33.816913 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:33.818206 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:33.818215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:33.818222 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:33.818247 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:33.818310 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:33.818339 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:33.818954 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:33.819044 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 02:51:33.819118 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:51:33.820199 21954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:33.820873 21964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [term 1 LEADER]: Becoming Leader. State: Replica: 417fa3a53e9e4fa98bfe1dd3693d22e3, State: Running, Role: LEADER
I20260929 02:51:33.821550 21964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260929 02:51:33.825058 21965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } } }
I20260929 02:51:33.825035 21966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417fa3a53e9e4fa98bfe1dd3693d22e3. Latest consensus state: current_term: 1 leader_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417fa3a53e9e4fa98bfe1dd3693d22e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } } }
I20260929 02:51:33.825526 21965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:33.825548 21966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3 [sys.catalog]: This master's current role is: LEADER
W20260929 02:51:33.831868 21980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:51:33.832044 21980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:51:33.832201 21976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:33.833968 21976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:33.842559 21976 catalog_manager.cc:1347] Generated new cluster ID: 839106c924f54b3c8c773662ec2fe845
I20260929 02:51:33.842710 21976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:33.854247 21976 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:33.856194 21976 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:33.867595 21976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 417fa3a53e9e4fa98bfe1dd3693d22e3: Generated new TSK 0
I20260929 02:51:33.868811 21976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:33.875761 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21887
W20260929 02:51:33.877090 21949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:40159
W20260929 02:51:33.877978 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21887: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:33.878151 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21887
W20260929 02:51:33.894965 21872 connection.cc:535] client connection to 127.0.0.1:44963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44963): Connection reset by peer (error 104)
2026-09-29T02:51:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1958 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:33 dist-test-slave-6tjh krb5kdc[21985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:33 dist-test-slave-6tjh krb5kdc[21985](info): set up 2 sockets
Sep 29 02:51:33 dist-test-slave-6tjh krb5kdc[21985](info): commencing operation
krb5kdc: starting...
W20260929 02:51:35.017098 388 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:35 dist-test-slave-6tjh krb5kdc[21985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 02:51:35.291635 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:35.653492 21998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:35.653818 21998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:35.654016 21998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:35.662128 21998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:35.662266 21998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:35.662393 21998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:35.662501 21998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:35.662585 21998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:35.672955 21998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41009
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:35.674342 21998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:35.675704 21998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:35.688149 22003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:35.688171 22004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:35.688334 22006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:35.688983 21998 server_base.cc:1034] running on GCE node
I20260929 02:51:35.689877 21998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:35.691402 21998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:35.692688 21998 hybrid_clock.cc:648] HybridClock initialized: now 1790650295692619 us; error 56 us; skew 500 ppm
I20260929 02:51:35.693115 21998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:35 dist-test-slave-6tjh krb5kdc[21985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650295, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:35.696427 21998 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 02:51:35.698246 21998 webserver.cc:460] Webserver started at http://127.0.0.1:37597/ using document root <none> and password file <none>
I20260929 02:51:35.698973 21998 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:35.699132 21998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:35.699440 21998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:35.702498 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "84ae2e2c1fdd44a892e873681cdbd85c"
format_stamp: "Formatted at 2026-09-29 02:51:35 on dist-test-slave-6tjh"
server_key: "a848e068beca95304d319cd2dedaf4d6"
server_key_iv: "7c51638be9c2734c7d53950a64788ca3"
server_key_version: "encryptionkey@0"
I20260929 02:51:35.703248 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "84ae2e2c1fdd44a892e873681cdbd85c"
format_stamp: "Formatted at 2026-09-29 02:51:35 on dist-test-slave-6tjh"
server_key: "a848e068beca95304d319cd2dedaf4d6"
server_key_iv: "7c51638be9c2734c7d53950a64788ca3"
server_key_version: "encryptionkey@0"
I20260929 02:51:35.708698 21998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:51:35.712662 22013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:35.714355 21998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:51:35.714587 21998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "84ae2e2c1fdd44a892e873681cdbd85c"
format_stamp: "Formatted at 2026-09-29 02:51:35 on dist-test-slave-6tjh"
server_key: "a848e068beca95304d319cd2dedaf4d6"
server_key_iv: "7c51638be9c2734c7d53950a64788ca3"
server_key_version: "encryptionkey@0"
I20260929 02:51:35.714825 21998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:35.721397 21998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:35.727397 21998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:35.727798 21998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:35.749115 21998 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41009
I20260929 02:51:35.749218 22064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41009 every 8 connection(s)
I20260929 02:51:35.750819 21998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:35.751242 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 21998
I20260929 02:51:35.751446 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:35.752175 388 external_mini_cluster.cc:1428] Setting key 8262ca4294e0bf1a671bb6f8f4f0defc
I20260929 02:51:35.755870 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:35.761672 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:35.754198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:57089 (local address 127.0.0.1:41009)
0929 02:51:35.755015 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:35.755031 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:35.755087 (+ 56us) server_negotiation.cc:407] Connection header received
0929 02:51:35.757518 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:35.757561 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:35.758013 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:35.758488 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:35.759566 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:35.759580 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:35.759599 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:35.759622 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:35.759712 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:35.759743 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:35.759977 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:35.760111 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:51:35.760317 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":345,"threads_started":1}
I20260929 02:51:35.766227 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:35.771174 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c: Bootstrap starting.
I20260929 02:51:35.774232 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:35.771563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.82:33789 (local address 127.0.0.1:41009)
0929 02:51:35.771805 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:35.771815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:35.772223 (+ 408us) server_negotiation.cc:407] Connection header received
0929 02:51:35.772394 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:35.772398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:35.772451 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:35.772534 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:35.773541 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:35.773545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:35.773548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:35.773566 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:35.773605 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:35.773620 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:35.773719 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:35.773857 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 02:51:35.773909 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:51:35.775637 22065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:35.776811 22065 log.cc:824] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:35.779803 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c: No bootstrap required, opened a new log
I20260929 02:51:35.787533 22065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260929 02:51:35.787967 22065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:35.788080 22065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84ae2e2c1fdd44a892e873681cdbd85c, State: Initialized, Role: FOLLOWER
I20260929 02:51:35.788738 22065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260929 02:51:35.788985 22065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:35.789130 22065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:35.789304 22065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:35.791482 22065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260929 02:51:35.791996 22065 leader_election.cc:302] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84ae2e2c1fdd44a892e873681cdbd85c; no voters:
I20260929 02:51:35.792723 22065 leader_election.cc:288] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:35.792902 22075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:35.793788 22075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [term 1 LEADER]: Becoming Leader. State: Replica: 84ae2e2c1fdd44a892e873681cdbd85c, State: Running, Role: LEADER
I20260929 02:51:35.794371 22075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } }
I20260929 02:51:35.794950 22065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:35.797580 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84ae2e2c1fdd44a892e873681cdbd85c. Latest consensus state: current_term: 1 leader_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } } }
I20260929 02:51:35.797948 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84ae2e2c1fdd44a892e873681cdbd85c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41009 } } }
I20260929 02:51:35.798180 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:35.798228 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:35.801266 22084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:35.806849 22084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:35.815671 22084 catalog_manager.cc:1347] Generated new cluster ID: adc8966482c4494abacce737be68bd8a
I20260929 02:51:35.815809 22084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:35.842049 22084 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:35.842641 22084 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:35.862567 22084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84ae2e2c1fdd44a892e873681cdbd85c: Generated new TSK 0
I20260929 02:51:35.863703 22084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:35.873528 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21998
W20260929 02:51:35.874703 22060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.82:57089
W20260929 02:51:35.875448 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 21998: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:35.875607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 21998
2026-09-29T02:51:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:35 dist-test-slave-6tjh krb5kdc[22096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:35 dist-test-slave-6tjh krb5kdc[22096](info): set up 2 sockets
Sep 29 02:51:35 dist-test-slave-6tjh krb5kdc[22096](info): commencing operation
krb5kdc: starting...
W20260929 02:51:37.057024 388 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.012s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:37 dist-test-slave-6tjh krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:37.334282 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42833
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:37.678136 22109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:37.678531 22109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:37.678730 22109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:37.686980 22109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:37.687177 22109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:37.687304 22109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:37.687431 22109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:37.687531 22109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:37.698406 22109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:37.699810 22109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:37.701277 22109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:37.713944 22115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:37.714032 22117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:37.714525 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:37.714835 22109 server_base.cc:1034] running on GCE node
I20260929 02:51:37.715734 22109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:37.717474 22109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:37.718745 22109 hybrid_clock.cc:648] HybridClock initialized: now 1790650297718701 us; error 42 us; skew 500 ppm
I20260929 02:51:37.719229 22109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:37 dist-test-slave-6tjh krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650297, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:37.722775 22109 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:37.724880 22109 webserver.cc:460] Webserver started at http://127.0.97.62:45583/ using document root <none> and password file <none>
I20260929 02:51:37.725934 22109 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:37.726092 22109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:37.726461 22109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:37.730921 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "fb0fd7caad5e4584afa20fec6d533cee"
format_stamp: "Formatted at 2026-09-29 02:51:37 on dist-test-slave-6tjh"
server_key: "ebd959f844df4e80ada3209d5536da23"
server_key_iv: "2c532b5ce73d8e070f8a7fff5662cf40"
server_key_version: "encryptionkey@0"
I20260929 02:51:37.732031 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "fb0fd7caad5e4584afa20fec6d533cee"
format_stamp: "Formatted at 2026-09-29 02:51:37 on dist-test-slave-6tjh"
server_key: "ebd959f844df4e80ada3209d5536da23"
server_key_iv: "2c532b5ce73d8e070f8a7fff5662cf40"
server_key_version: "encryptionkey@0"
I20260929 02:51:37.737883 22109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 02:51:37.741479 22124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:37.742903 22109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 02:51:37.743046 22109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "fb0fd7caad5e4584afa20fec6d533cee"
format_stamp: "Formatted at 2026-09-29 02:51:37 on dist-test-slave-6tjh"
server_key: "ebd959f844df4e80ada3209d5536da23"
server_key_iv: "2c532b5ce73d8e070f8a7fff5662cf40"
server_key_version: "encryptionkey@0"
I20260929 02:51:37.743213 22109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:37.754957 22109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:37.759485 22109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:37.759822 22109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:37.780835 22109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42833
I20260929 02:51:37.780889 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42833 every 8 connection(s)
I20260929 02:51:37.782405 22109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:37.786582 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:37.790686 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22109
I20260929 02:51:37.790884 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:37.791477 388 external_mini_cluster.cc:1428] Setting key c1f373d26ef564aa87890ab77f1cf009
I20260929 02:51:37.800817 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee: Bootstrap starting.
I20260929 02:51:37.805166 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:37.806795 22176 log.cc:824] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:51:37 dist-test-slave-6tjh krb5kdc[22096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:37.810882 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee: No bootstrap required, opened a new log
I20260929 02:51:37.819242 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:37.793439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59805 (local address 127.0.97.62:42833)
0929 02:51:37.794295 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:37.794307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:37.794348 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:51:37.796033 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:37.796063 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:37.796500 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:37.796932 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:37.798956 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:37.800421 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:37.802476 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:37.803358 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:37.803550 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:37.809751 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:37.809761 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:37.809782 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:37.809825 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:37.813693 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:37.814901 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:37.814910 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:37.814925 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:37.815191 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:37.815782 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:37.815786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:37.815789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:37.816483 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:37.816875 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:37.817466 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 02:51:37.817939 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":420,"threads_started":1}
I20260929 02:51:37.820601 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0fd7caad5e4584afa20fec6d533cee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42833 } }
I20260929 02:51:37.821038 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:37.821152 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0fd7caad5e4584afa20fec6d533cee, State: Initialized, Role: FOLLOWER
I20260929 02:51:37.821877 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0fd7caad5e4584afa20fec6d533cee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42833 } }
I20260929 02:51:37.822149 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:37.822311 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:37.822508 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:37.824136 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:37.825218 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0fd7caad5e4584afa20fec6d533cee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42833 } }
I20260929 02:51:37.825750 22176 leader_election.cc:302] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0fd7caad5e4584afa20fec6d533cee; no voters:
I20260929 02:51:37.826412 22176 leader_election.cc:288] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:37.826634 22181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:37.828536 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:37.829170 22181 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [term 1 LEADER]: Becoming Leader. State: Replica: fb0fd7caad5e4584afa20fec6d533cee, State: Running, Role: LEADER
I20260929 02:51:37.830240 22181 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0fd7caad5e4584afa20fec6d533cee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42833 } }
I20260929 02:51:37.833770 22184 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb0fd7caad5e4584afa20fec6d533cee. Latest consensus state: current_term: 1 leader_uuid: "fb0fd7caad5e4584afa20fec6d533cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0fd7caad5e4584afa20fec6d533cee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42833 } } }
I20260929 02:51:37.834275 22184 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:37.835023 22182 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb0fd7caad5e4584afa20fec6d533cee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0fd7caad5e4584afa20fec6d533cee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42833 } } }
I20260929 02:51:37.835363 22182 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:37.858170 22196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:37.867362 22196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:37.868058 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:37.833764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33877 (local address 127.0.97.62:42833)
0929 02:51:37.838860 (+ 5096us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:37.838871 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:37.843769 (+ 4898us) server_negotiation.cc:407] Connection header received
0929 02:51:37.844425 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:37.844436 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:37.844579 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:37.844684 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:37.847912 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:37.849740 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:37.853063 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:37.854210 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:37.854358 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:37.858737 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:37.858745 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:37.858750 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:37.858798 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:37.863481 (+ 4683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:37.864784 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:37.864792 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:37.864796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:37.865100 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:37.865768 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:37.865775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:37.865779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:37.866281 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:37.866561 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:37.867393 (+ 832us) server_negotiation.cc:299] Negotiation successful
0929 02:51:37.867620 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4905}
I20260929 02:51:37.881031 22196 catalog_manager.cc:1347] Generated new cluster ID: b692a9a709f84940911fe550e1a1603e
I20260929 02:51:37.881177 22196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:37.896474 22196 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:37.898545 22196 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:37.910969 22196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb0fd7caad5e4584afa20fec6d533cee: Generated new TSK 0
I20260929 02:51:37.911872 22196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:37.919486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22109
I20260929 02:51:37.920789 22171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:38.014071 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22109
W20260929 02:51:38.031513 22094 connection.cc:535] client connection to 127.0.97.62:42833 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42833): Connection reset by peer (error 104)
2026-09-29T02:51:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2137 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:38 dist-test-slave-6tjh krb5kdc[22209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:38 dist-test-slave-6tjh krb5kdc[22209](info): set up 2 sockets
Sep 29 02:51:38 dist-test-slave-6tjh krb5kdc[22209](info): commencing operation
krb5kdc: starting...
W20260929 02:51:39.185045 388 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:39 dist-test-slave-6tjh krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:39.465010 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40941
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:39.818086 22222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:39.818390 22222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:39.818485 22222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:39.826027 22222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:39.826149 22222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:39.826196 22222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:39.826257 22222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:39.826300 22222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:39.836640 22222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40941
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:39.837893 22222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:39.839190 22222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:39.851189 22230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:39.851255 22228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:39.851413 22227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:39.852340 22222 server_base.cc:1034] running on GCE node
I20260929 02:51:39.853156 22222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:39.854512 22222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:39.855743 22222 hybrid_clock.cc:648] HybridClock initialized: now 1790650299855694 us; error 32 us; skew 500 ppm
I20260929 02:51:39.856154 22222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:39 dist-test-slave-6tjh krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650299, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:39.865290 22222 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:39.867080 22222 webserver.cc:460] Webserver started at http://127.0.97.62:36033/ using document root <none> and password file <none>
I20260929 02:51:39.867837 22222 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:39.867990 22222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:39.868476 22222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:39.871647 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "64f69ee5b0584b6480e456814cc995c7"
format_stamp: "Formatted at 2026-09-29 02:51:39 on dist-test-slave-6tjh"
server_key: "a7bc3aeb8ba0a360095005614054f9dc"
server_key_iv: "181ef85b66d1beec0d43ce1bcf441fbf"
server_key_version: "encryptionkey@0"
I20260929 02:51:39.872423 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "64f69ee5b0584b6480e456814cc995c7"
format_stamp: "Formatted at 2026-09-29 02:51:39 on dist-test-slave-6tjh"
server_key: "a7bc3aeb8ba0a360095005614054f9dc"
server_key_iv: "181ef85b66d1beec0d43ce1bcf441fbf"
server_key_version: "encryptionkey@0"
I20260929 02:51:39.877750 22222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 02:51:39.881606 22237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:39.883244 22222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:51:39.883445 22222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "64f69ee5b0584b6480e456814cc995c7"
format_stamp: "Formatted at 2026-09-29 02:51:39 on dist-test-slave-6tjh"
server_key: "a7bc3aeb8ba0a360095005614054f9dc"
server_key_iv: "181ef85b66d1beec0d43ce1bcf441fbf"
server_key_version: "encryptionkey@0"
I20260929 02:51:39.883664 22222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:39.901268 22222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:39.907707 22222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:39.908054 22222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:39.927085 22222 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40941
I20260929 02:51:39.927106 22288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40941 every 8 connection(s)
I20260929 02:51:39.928723 22222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:39.932199 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22222
I20260929 02:51:39.932405 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:39.933036 388 external_mini_cluster.cc:1428] Setting key 8d9610c1a18a894a237a2f4b6a7ed3f6
I20260929 02:51:39.934020 22289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:39.946521 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:39.935046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54133 (local address 127.0.97.62:40941)
0929 02:51:39.935867 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:39.935879 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:39.935920 (+ 41us) server_negotiation.cc:407] Connection header received
0929 02:51:39.937621 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:39.937643 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:39.938068 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:39.938503 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:39.939056 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:39.940413 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:39.941794 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:39.942482 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:39.942684 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:39.944104 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:39.944113 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:39.944128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:39.944803 (+ 675us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:39.944883 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:39.944907 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:39.945086 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:39.945373 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 02:51:39.945551 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":398,"threads_started":1}
I20260929 02:51:39.947149 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7: Bootstrap starting.
I20260929 02:51:39.950891 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:39.951146 22289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:39.952184 22289 log.cc:824] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:39.955358 22289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7: No bootstrap required, opened a new log
I20260929 02:51:39.961609 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:39.955122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32833 (local address 127.0.97.62:40941)
0929 02:51:39.955300 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:39.955308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:39.955847 (+ 539us) server_negotiation.cc:407] Connection header received
0929 02:51:39.956078 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:39.956082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:39.956139 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:39.956251 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:39.956772 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:39.957590 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:39.958945 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:39.959435 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:39.959599 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:39.960871 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:39.960876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:39.960879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:39.960898 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:39.960940 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:39.960955 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:39.961093 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:39.961296 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 02:51:39.961354 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:51:39.964298 22289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f69ee5b0584b6480e456814cc995c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260929 02:51:39.964793 22289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:39.964941 22289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64f69ee5b0584b6480e456814cc995c7, State: Initialized, Role: FOLLOWER
I20260929 02:51:39.965690 22289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f69ee5b0584b6480e456814cc995c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260929 02:51:39.965900 22289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:39.966006 22289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:39.966132 22289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:39.968369 22289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f69ee5b0584b6480e456814cc995c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260929 02:51:39.968986 22289 leader_election.cc:302] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64f69ee5b0584b6480e456814cc995c7; no voters:
I20260929 02:51:39.969822 22289 leader_election.cc:288] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:39.969957 22299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:39.970669 22299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [term 1 LEADER]: Becoming Leader. State: Replica: 64f69ee5b0584b6480e456814cc995c7, State: Running, Role: LEADER
I20260929 02:51:39.971204 22299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f69ee5b0584b6480e456814cc995c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } }
I20260929 02:51:39.971982 22289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:39.974332 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64f69ee5b0584b6480e456814cc995c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f69ee5b0584b6480e456814cc995c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } } }
I20260929 02:51:39.974735 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:39.974320 22301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64f69ee5b0584b6480e456814cc995c7. Latest consensus state: current_term: 1 leader_uuid: "64f69ee5b0584b6480e456814cc995c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64f69ee5b0584b6480e456814cc995c7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40941 } } }
I20260929 02:51:39.975281 22301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:39.982509 22304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:39.987269 22304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:39.996006 22304 catalog_manager.cc:1347] Generated new cluster ID: 029d003c35994f0d8fe69cbce0f1db7b
I20260929 02:51:39.996143 22304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:40.020056 22304 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:40.022149 22304 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:40.032886 22304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64f69ee5b0584b6480e456814cc995c7: Generated new TSK 0
I20260929 02:51:40.034123 22304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:40.042481 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22222
W20260929 02:51:40.044432 22284 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54133
W20260929 02:51:40.045434 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22222: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:40.045615 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22222
W20260929 02:51:40.064875 22207 connection.cc:535] client connection to 127.0.97.62:40941 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:40941): Connection reset by peer (error 104)
2026-09-29T02:51:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:40 dist-test-slave-6tjh krb5kdc[22320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:40 dist-test-slave-6tjh krb5kdc[22320](info): set up 2 sockets
Sep 29 02:51:40 dist-test-slave-6tjh krb5kdc[22320](info): commencing operation
krb5kdc: starting...
W20260929 02:51:41.181084 388 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:41 dist-test-slave-6tjh krb5kdc[22320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:41.456097 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45307
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45307
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:41.809981 22333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:41.810292 22333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:41.810448 22333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:41.817956 22333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:41.818089 22333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:41.818199 22333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:41.818298 22333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:41.818375 22333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:41.828444 22333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45307
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:41.829864 22333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:41.831249 22333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:41.844089 22339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:41.844214 22338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:41.844192 22341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:41.844975 22333 server_base.cc:1034] running on GCE node
I20260929 02:51:41.845768 22333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:41.847304 22333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:41.848659 22333 hybrid_clock.cc:648] HybridClock initialized: now 1790650301848581 us; error 63 us; skew 500 ppm
I20260929 02:51:41.849104 22333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:41 dist-test-slave-6tjh krb5kdc[22320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:41.852931 22333 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:41.854707 22333 webserver.cc:460] Webserver started at http://127.0.97.62:40389/ using document root <none> and password file <none>
I20260929 02:51:41.855501 22333 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:41.855662 22333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:41.856003 22333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:41.859653 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "361495b9455c4efebeeb70aee49e8f51"
format_stamp: "Formatted at 2026-09-29 02:51:41 on dist-test-slave-6tjh"
server_key: "9dcf1db6b5f0d4b94c12c990c900bbf2"
server_key_iv: "d4176b4a6abdb6d3c6f1351d1792fee9"
server_key_version: "encryptionkey@0"
I20260929 02:51:41.860538 22333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "361495b9455c4efebeeb70aee49e8f51"
format_stamp: "Formatted at 2026-09-29 02:51:41 on dist-test-slave-6tjh"
server_key: "9dcf1db6b5f0d4b94c12c990c900bbf2"
server_key_iv: "d4176b4a6abdb6d3c6f1351d1792fee9"
server_key_version: "encryptionkey@0"
I20260929 02:51:41.866354 22333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:51:41.870352 22348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:41.872087 22333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:51:41.872316 22333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "361495b9455c4efebeeb70aee49e8f51"
format_stamp: "Formatted at 2026-09-29 02:51:41 on dist-test-slave-6tjh"
server_key: "9dcf1db6b5f0d4b94c12c990c900bbf2"
server_key_iv: "d4176b4a6abdb6d3c6f1351d1792fee9"
server_key_version: "encryptionkey@0"
I20260929 02:51:41.872613 22333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:41.879410 22333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:41.884299 22333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:41.884687 22333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:41.904709 22333 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45307
I20260929 02:51:41.904763 22399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45307 every 8 connection(s)
I20260929 02:51:41.906201 22333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:41.910111 22400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:41.911540 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22333
I20260929 02:51:41.911728 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:41.912201 388 external_mini_cluster.cc:1428] Setting key b7e5379c9fdafe936638e3bae32a91d8
I20260929 02:51:41.920384 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:41.914171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42277 (local address 127.0.97.62:45307)
0929 02:51:41.914998 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:41.915011 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:41.915105 (+ 94us) server_negotiation.cc:407] Connection header received
0929 02:51:41.916775 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:41.916805 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:41.917182 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:41.917518 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:41.918658 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:41.918665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:41.918679 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:41.918696 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:41.918768 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:41.918795 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:41.918986 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:41.919107 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 02:51:41.919341 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":382,"threads_started":1}
I20260929 02:51:41.923833 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51: Bootstrap starting.
I20260929 02:51:41.924458 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:41.926949 22400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:41.928108 22400 log.cc:824] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:41.931627 22403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:41.929063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35039 (local address 127.0.97.62:45307)
0929 02:51:41.929295 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:41.929306 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:41.929732 (+ 426us) server_negotiation.cc:407] Connection header received
0929 02:51:41.929894 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:41.929899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:41.929965 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:41.930079 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:41.930975 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:41.930981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:41.930986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:41.931006 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:41.931053 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:41.931073 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:41.931207 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:41.931255 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 02:51:41.931317 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 02:51:41.931831 22400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51: No bootstrap required, opened a new log
I20260929 02:51:41.940122 22400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361495b9455c4efebeeb70aee49e8f51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45307 } }
I20260929 02:51:41.940580 22400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:41.940708 22400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361495b9455c4efebeeb70aee49e8f51, State: Initialized, Role: FOLLOWER
I20260929 02:51:41.941352 22400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361495b9455c4efebeeb70aee49e8f51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45307 } }
I20260929 02:51:41.941613 22400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:41.941776 22400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:41.941972 22400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:41.944402 22400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361495b9455c4efebeeb70aee49e8f51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45307 } }
I20260929 02:51:41.944959 22400 leader_election.cc:302] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 361495b9455c4efebeeb70aee49e8f51; no voters:
I20260929 02:51:41.945588 22400 leader_election.cc:288] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:41.945760 22410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:41.946408 22410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [term 1 LEADER]: Becoming Leader. State: Replica: 361495b9455c4efebeeb70aee49e8f51, State: Running, Role: LEADER
I20260929 02:51:41.946979 22410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361495b9455c4efebeeb70aee49e8f51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45307 } }
I20260929 02:51:41.948112 22400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:41.950129 22412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 361495b9455c4efebeeb70aee49e8f51. Latest consensus state: current_term: 1 leader_uuid: "361495b9455c4efebeeb70aee49e8f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361495b9455c4efebeeb70aee49e8f51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45307 } } }
I20260929 02:51:41.950603 22412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:41.950096 22411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "361495b9455c4efebeeb70aee49e8f51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361495b9455c4efebeeb70aee49e8f51" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45307 } } }
I20260929 02:51:41.951145 22411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:41.952730 22417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:41.958258 22417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:41.968242 22417 catalog_manager.cc:1347] Generated new cluster ID: eec814d5b81240cb8eef7c5e0573dcb8
I20260929 02:51:41.968375 22417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:41.991287 22417 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:41.991859 22417 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:41.998358 22417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 361495b9455c4efebeeb70aee49e8f51: Generated new TSK 0
I20260929 02:51:41.999389 22417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:42.007782 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22333
W20260929 02:51:42.008909 22395 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42277
W20260929 02:51:42.009673 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22333: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:42.009832 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22333
2026-09-29T02:51:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:42 dist-test-slave-6tjh krb5kdc[22431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:42 dist-test-slave-6tjh krb5kdc[22431](info): set up 2 sockets
Sep 29 02:51:42 dist-test-slave-6tjh krb5kdc[22431](info): commencing operation
krb5kdc: starting...
W20260929 02:51:43.141119 388 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:43 dist-test-slave-6tjh krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:43.412781 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44451
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:43.753080 22444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:43.753396 22444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:43.753508 22444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:43.761814 22444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:43.761950 22444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:43.762005 22444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:43.762069 22444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:43.762125 22444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:43.772285 22444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:43.773649 22444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:43.774930 22444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:43.787078 22450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:43.787159 22449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:43.787129 22452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:43.788484 22444 server_base.cc:1034] running on GCE node
I20260929 02:51:43.789414 22444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:43.790997 22444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:43.792232 22444 hybrid_clock.cc:648] HybridClock initialized: now 1790650303792202 us; error 43 us; skew 500 ppm
I20260929 02:51:43.792666 22444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:43 dist-test-slave-6tjh krb5kdc[22431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:43.795955 22444 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:43.797729 22444 webserver.cc:460] Webserver started at http://127.0.97.62:36887/ using document root <none> and password file <none>
I20260929 02:51:43.798525 22444 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:43.798686 22444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:43.799006 22444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:43.802330 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "77bf6c75ac9641afb3109fb431173c9f"
format_stamp: "Formatted at 2026-09-29 02:51:43 on dist-test-slave-6tjh"
server_key: "55018bce161e03230d5a45de51979107"
server_key_iv: "80e26e169f6792d5699a79b1367d7e78"
server_key_version: "encryptionkey@0"
I20260929 02:51:43.803165 22444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "77bf6c75ac9641afb3109fb431173c9f"
format_stamp: "Formatted at 2026-09-29 02:51:43 on dist-test-slave-6tjh"
server_key: "55018bce161e03230d5a45de51979107"
server_key_iv: "80e26e169f6792d5699a79b1367d7e78"
server_key_version: "encryptionkey@0"
I20260929 02:51:43.808959 22444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 02:51:43.813040 22459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:43.814586 22444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:51:43.814803 22444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "77bf6c75ac9641afb3109fb431173c9f"
format_stamp: "Formatted at 2026-09-29 02:51:43 on dist-test-slave-6tjh"
server_key: "55018bce161e03230d5a45de51979107"
server_key_iv: "80e26e169f6792d5699a79b1367d7e78"
server_key_version: "encryptionkey@0"
I20260929 02:51:43.815034 22444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:43.832466 22444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:43.836839 22444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:43.837148 22444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:43.856324 22444 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44451
I20260929 02:51:43.856355 22510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44451 every 8 connection(s)
I20260929 02:51:43.857843 22444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:43.858561 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22444
I20260929 02:51:43.858771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:43.859402 388 external_mini_cluster.cc:1428] Setting key 7f2ba1e43c34290927706ff47bbdbb2d
I20260929 02:51:43.862380 22511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:51:43 dist-test-slave-6tjh krb5kdc[22431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:43.875782 22511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f: Bootstrap starting.
I20260929 02:51:43.879189 22511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:43.880280 22511 log.cc:824] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:43.883778 22511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f: No bootstrap required, opened a new log
I20260929 02:51:43.883873 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:43.861321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49499 (local address 127.0.97.62:44451)
0929 02:51:43.862220 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:43.862233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:43.862276 (+ 43us) server_negotiation.cc:407] Connection header received
0929 02:51:43.864126 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:43.864150 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:43.864610 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:43.864994 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:43.866801 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:43.868326 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:43.870164 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:43.870862 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:43.871005 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:43.875997 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:43.876006 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:43.876021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:43.876061 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:43.879159 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:43.879888 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:43.879895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:43.879902 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:43.880077 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:43.880539 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:43.880577 (+ 38us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:43.880582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:43.881283 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:43.881522 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:43.882069 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 02:51:43.882506 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":467,"threads_started":1}
I20260929 02:51:43.887607 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:43.891794 22511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bf6c75ac9641afb3109fb431173c9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44451 } }
I20260929 02:51:43.892248 22511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:43.892416 22511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77bf6c75ac9641afb3109fb431173c9f, State: Initialized, Role: FOLLOWER
I20260929 02:51:43.893186 22511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bf6c75ac9641afb3109fb431173c9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44451 } }
I20260929 02:51:43.893513 22511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:43.893680 22511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:43.893913 22511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:43.896731 22511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bf6c75ac9641afb3109fb431173c9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44451 } }
I20260929 02:51:43.897295 22511 leader_election.cc:302] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77bf6c75ac9641afb3109fb431173c9f; no voters:
I20260929 02:51:43.897912 22511 leader_election.cc:288] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:43.898082 22521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:43.899449 22521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [term 1 LEADER]: Becoming Leader. State: Replica: 77bf6c75ac9641afb3109fb431173c9f, State: Running, Role: LEADER
I20260929 02:51:43.900381 22511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:43.900203 22521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bf6c75ac9641afb3109fb431173c9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44451 } }
I20260929 02:51:43.904878 22523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77bf6c75ac9641afb3109fb431173c9f. Latest consensus state: current_term: 1 leader_uuid: "77bf6c75ac9641afb3109fb431173c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bf6c75ac9641afb3109fb431173c9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44451 } } }
I20260929 02:51:43.905289 22523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:43.904383 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77bf6c75ac9641afb3109fb431173c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77bf6c75ac9641afb3109fb431173c9f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44451 } } }
I20260929 02:51:43.905905 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:43.913523 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:43.892538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39299 (local address 127.0.97.62:44451)
0929 02:51:43.892763 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:43.892772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:43.893250 (+ 478us) server_negotiation.cc:407] Connection header received
0929 02:51:43.893402 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:43.893407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:43.893500 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:43.893616 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:43.895398 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:43.896515 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:43.898416 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:43.899440 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:43.899577 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:43.903530 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:43.903541 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:43.903546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:43.903604 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:43.908492 (+ 4888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:43.909903 (+ 1411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:43.909913 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:43.909921 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:43.910234 (+ 313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:43.910854 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:43.910863 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:43.910870 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:43.911445 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:43.911793 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:43.912595 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 02:51:43.912941 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:51:43.918653 22530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:43.924783 22530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:43.934748 22530 catalog_manager.cc:1347] Generated new cluster ID: 5fbfb3894da14db8bb5fa9a516e9d9f8
I20260929 02:51:43.934898 22530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:43.953120 22530 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:43.955128 22530 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:43.962453 22530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77bf6c75ac9641afb3109fb431173c9f: Generated new TSK 0
I20260929 02:51:43.963620 22530 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:43.979827 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22444
I20260929 02:51:43.981170 22506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:44.075672 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22444
2026-09-29T02:51:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:44 dist-test-slave-6tjh krb5kdc[22544](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:44 dist-test-slave-6tjh krb5kdc[22544](info): set up 2 sockets
Sep 29 02:51:44 dist-test-slave-6tjh krb5kdc[22544](info): commencing operation
krb5kdc: starting...
W20260929 02:51:45.205040 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:45 dist-test-slave-6tjh krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:45.480206 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33935
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:45.824225 22557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:45.824621 22557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:45.824780 22557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:45.832198 22557 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:45.832368 22557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:45.832484 22557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:45.832598 22557 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:45.832680 22557 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:45.842809 22557 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:45.844166 22557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:45.845547 22557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:45.860841 22562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:45.860916 22565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:45.860884 22563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:45.862205 22557 server_base.cc:1034] running on GCE node
I20260929 02:51:45.863041 22557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:45.864605 22557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:45.865840 22557 hybrid_clock.cc:648] HybridClock initialized: now 1790650305865784 us; error 57 us; skew 500 ppm
I20260929 02:51:45.866204 22557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:45 dist-test-slave-6tjh krb5kdc[22544](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:45.869565 22557 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:45.871208 22557 webserver.cc:460] Webserver started at http://127.0.97.62:32971/ using document root <none> and password file <none>
I20260929 02:51:45.871914 22557 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:45.872040 22557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:45.872362 22557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:45.875526 22557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "ce1698d1922c469cae045306244b8fff"
format_stamp: "Formatted at 2026-09-29 02:51:45 on dist-test-slave-6tjh"
server_key: "6512bc182cc5e7a8028fb9cbd193df13"
server_key_iv: "3ebaa208570895057306e166ff0e17d5"
server_key_version: "encryptionkey@0"
I20260929 02:51:45.876389 22557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "ce1698d1922c469cae045306244b8fff"
format_stamp: "Formatted at 2026-09-29 02:51:45 on dist-test-slave-6tjh"
server_key: "6512bc182cc5e7a8028fb9cbd193df13"
server_key_iv: "3ebaa208570895057306e166ff0e17d5"
server_key_version: "encryptionkey@0"
I20260929 02:51:45.882438 22557 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 02:51:45.886160 22572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:45.887694 22557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:51:45.887887 22557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "ce1698d1922c469cae045306244b8fff"
format_stamp: "Formatted at 2026-09-29 02:51:45 on dist-test-slave-6tjh"
server_key: "6512bc182cc5e7a8028fb9cbd193df13"
server_key_iv: "3ebaa208570895057306e166ff0e17d5"
server_key_version: "encryptionkey@0"
I20260929 02:51:45.888120 22557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:45.926229 22557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:45.930995 22557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:45.931344 22557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:45.950536 22557 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33935
I20260929 02:51:45.950573 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33935 every 8 connection(s)
I20260929 02:51:45.951924 22557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:45.955869 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:45.957461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22557
I20260929 02:51:45.957655 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:45.958140 388 external_mini_cluster.cc:1428] Setting key 4f38963206efcd8228a593e1fbb9f539
I20260929 02:51:45.970070 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff: Bootstrap starting.
I20260929 02:51:45.972765 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:45.960045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47963 (local address 127.0.97.62:33935)
0929 02:51:45.960988 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:45.961008 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:45.961060 (+ 52us) server_negotiation.cc:407] Connection header received
0929 02:51:45.962726 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:45.962759 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:45.963318 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:45.963771 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:45.964244 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:45.966147 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:45.967448 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:45.968484 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:45.968683 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:45.969857 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:45.969869 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:45.969886 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:45.970600 (+ 714us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:45.970703 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:45.970734 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:45.970841 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:45.971124 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 02:51:45.971378 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":440,"threads_started":1}
I20260929 02:51:45.973910 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:45.975255 22624 log.cc:824] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:45.976908 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:45.978119 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff: No bootstrap required, opened a new log
I20260929 02:51:45.986215 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1698d1922c469cae045306244b8fff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33935 } }
I20260929 02:51:45.986619 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:45.986759 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce1698d1922c469cae045306244b8fff, State: Initialized, Role: FOLLOWER
I20260929 02:51:45.987408 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1698d1922c469cae045306244b8fff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33935 } }
I20260929 02:51:45.987685 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:45.987821 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:45.987990 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:45.990995 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1698d1922c469cae045306244b8fff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33935 } }
I20260929 02:51:45.991597 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:45.981122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37919 (local address 127.0.97.62:33935)
0929 02:51:45.981365 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:45.981374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:45.985167 (+ 3793us) server_negotiation.cc:407] Connection header received
0929 02:51:45.985339 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:45.985344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:45.985412 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:45.985515 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:45.986146 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:45.987009 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:45.988910 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:45.989424 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:45.989535 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:45.990762 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:45.990767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:45.990770 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:45.990787 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:45.990825 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:45.990839 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:45.990912 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:45.991125 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 02:51:45.991247 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 02:51:45.991622 22624 leader_election.cc:302] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce1698d1922c469cae045306244b8fff; no voters:
I20260929 02:51:45.992353 22624 leader_election.cc:288] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:45.992488 22634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:45.996767 22634 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [term 1 LEADER]: Becoming Leader. State: Replica: ce1698d1922c469cae045306244b8fff, State: Running, Role: LEADER
I20260929 02:51:45.997556 22634 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1698d1922c469cae045306244b8fff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33935 } }
I20260929 02:51:45.998792 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:46.000758 22636 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce1698d1922c469cae045306244b8fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1698d1922c469cae045306244b8fff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33935 } } }
I20260929 02:51:46.001232 22636 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:46.001718 22635 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce1698d1922c469cae045306244b8fff. Latest consensus state: current_term: 1 leader_uuid: "ce1698d1922c469cae045306244b8fff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce1698d1922c469cae045306244b8fff" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33935 } } }
I20260929 02:51:46.002105 22635 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:46.002259 22642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:46.008052 22642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:46.017508 22642 catalog_manager.cc:1347] Generated new cluster ID: 3099584f15c6447cb2519e9f93f68047
I20260929 02:51:46.017660 22642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:46.039410 22642 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:46.041519 22642 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:46.053779 22642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce1698d1922c469cae045306244b8fff: Generated new TSK 0
I20260929 02:51:46.054899 22642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:46.069330 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22557
W20260929 02:51:46.070550 22619 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47963
W20260929 02:51:46.071368 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22557: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:46.071516 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22557
2026-09-29T02:51:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:46 dist-test-slave-6tjh krb5kdc[22655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:46 dist-test-slave-6tjh krb5kdc[22655](info): set up 2 sockets
Sep 29 02:51:46 dist-test-slave-6tjh krb5kdc[22655](info): commencing operation
krb5kdc: starting...
W20260929 02:51:47.193045 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:47 dist-test-slave-6tjh krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:47.467273 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36451
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:47.809808 22668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:47.810141 22668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:47.810339 22668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:47.818795 22668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:47.818941 22668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:47.819123 22668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:47.819244 22668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:47.819331 22668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:47.830377 22668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:47.831707 22668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:47.833081 22668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:47.845314 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:47.845320 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:47.845367 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:47.846379 22668 server_base.cc:1034] running on GCE node
I20260929 02:51:47.847242 22668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:47.848913 22668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:47.850159 22668 hybrid_clock.cc:648] HybridClock initialized: now 1790650307850112 us; error 41 us; skew 500 ppm
I20260929 02:51:47.850569 22668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:47 dist-test-slave-6tjh krb5kdc[22655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650307, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:47.854120 22668 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:47.855782 22668 webserver.cc:460] Webserver started at http://127.0.97.62:44123/ using document root <none> and password file <none>
I20260929 02:51:47.856506 22668 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:47.856705 22668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:47.857019 22668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:47.860121 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "d61306456e26465a906d6c169a73c261"
format_stamp: "Formatted at 2026-09-29 02:51:47 on dist-test-slave-6tjh"
server_key: "d14688efecf1977b458b9cdaf2054896"
server_key_iv: "a0959e174413a1664447008cd4b73a2f"
server_key_version: "encryptionkey@0"
I20260929 02:51:47.860981 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "d61306456e26465a906d6c169a73c261"
format_stamp: "Formatted at 2026-09-29 02:51:47 on dist-test-slave-6tjh"
server_key: "d14688efecf1977b458b9cdaf2054896"
server_key_iv: "a0959e174413a1664447008cd4b73a2f"
server_key_version: "encryptionkey@0"
I20260929 02:51:47.866571 22668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:51:47.870394 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:47.871867 22668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:51:47.872066 22668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "d61306456e26465a906d6c169a73c261"
format_stamp: "Formatted at 2026-09-29 02:51:47 on dist-test-slave-6tjh"
server_key: "d14688efecf1977b458b9cdaf2054896"
server_key_iv: "a0959e174413a1664447008cd4b73a2f"
server_key_version: "encryptionkey@0"
I20260929 02:51:47.872283 22668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:47.878857 22668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:47.883344 22668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:47.883651 22668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:47.902796 22668 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36451
I20260929 02:51:47.902855 22734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36451 every 8 connection(s)
I20260929 02:51:47.904243 22668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:47.908291 22735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:47.913677 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22668
I20260929 02:51:47.913874 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:47.914453 388 external_mini_cluster.cc:1428] Setting key fb6ca2c5c6dbbd516fa1b6f0d82f62bc
I20260929 02:51:47.921316 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261: Bootstrap starting.
I20260929 02:51:47.922986 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:47.916387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44787 (local address 127.0.97.62:36451)
0929 02:51:47.917274 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:47.917286 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:47.917337 (+ 51us) server_negotiation.cc:407] Connection header received
0929 02:51:47.919396 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:47.919422 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:47.919788 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:47.920131 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:47.921356 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:47.921363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:47.921377 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:47.921393 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:47.921454 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:47.921508 (+ 54us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:47.921688 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:47.921822 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:51:47.922029 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":433,"threads_started":1}
I20260929 02:51:47.925407 22735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:47.926662 22735 log.cc:824] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:47.927177 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:47.929956 22735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261: No bootstrap required, opened a new log
I20260929 02:51:47.934630 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:47.932148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33263 (local address 127.0.97.62:36451)
0929 02:51:47.932337 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:47.932344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:47.932760 (+ 416us) server_negotiation.cc:407] Connection header received
0929 02:51:47.932909 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:47.932914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:47.932976 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:47.933097 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:47.933986 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:47.933992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:47.933999 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 02:51:47.934023 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:47.934071 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 02:51:47.934094 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:47.934228 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:47.934271 (+ 43us) server_negotiation.cc:299] Negotiation successful
0929 02:51:47.934328 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 02:51:47.938947 22735 raft_consensus.cc:348] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61306456e26465a906d6c169a73c261" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36451 } }
I20260929 02:51:47.939370 22735 raft_consensus.cc:374] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:51:47.939483 22735 raft_consensus.cc:729] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d61306456e26465a906d6c169a73c261, State: Initialized, Role: FOLLOWER
I20260929 02:51:47.940217 22735 consensus_queue.cc:260] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61306456e26465a906d6c169a73c261" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36451 } }
I20260929 02:51:47.940503 22735 raft_consensus.cc:388] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:51:47.940676 22735 raft_consensus.cc:482] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:51:47.940814 22735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:47.942965 22735 raft_consensus.cc:504] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61306456e26465a906d6c169a73c261" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36451 } }
I20260929 02:51:47.943418 22735 leader_election.cc:302] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d61306456e26465a906d6c169a73c261; no voters:
I20260929 02:51:47.943938 22735 leader_election.cc:288] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:51:47.944079 22745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:47.944777 22745 raft_consensus.cc:686] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [term 1 LEADER]: Becoming Leader. State: Replica: d61306456e26465a906d6c169a73c261, State: Running, Role: LEADER
I20260929 02:51:47.945344 22745 consensus_queue.cc:237] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61306456e26465a906d6c169a73c261" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36451 } }
I20260929 02:51:47.946216 22735 sys_catalog.cc:564] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:47.948328 22747 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d61306456e26465a906d6c169a73c261. Latest consensus state: current_term: 1 leader_uuid: "d61306456e26465a906d6c169a73c261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61306456e26465a906d6c169a73c261" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36451 } } }
I20260929 02:51:47.948253 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d61306456e26465a906d6c169a73c261" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d61306456e26465a906d6c169a73c261" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36451 } } }
I20260929 02:51:47.948715 22747 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:47.948733 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:47.949702 22750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:47.955900 22750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:47.965833 22750 catalog_manager.cc:1347] Generated new cluster ID: b998217856724fe28c3d21ae90bdbecf
I20260929 02:51:47.965973 22750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:47.977953 22750 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:47.978484 22750 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:47.984661 22750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d61306456e26465a906d6c169a73c261: Generated new TSK 0
I20260929 02:51:47.985671 22750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:48.001526 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22668
W20260929 02:51:48.002699 22730 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44787
W20260929 02:51:48.003417 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22668: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 02:51:48.003568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22668
2026-09-29T02:51:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1932 ms)
[----------] 18 tests from AuthTokenIssuingTest (36213 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:48 dist-test-slave-6tjh krb5kdc[22766](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:48 dist-test-slave-6tjh krb5kdc[22766](info): set up 2 sockets
Sep 29 02:51:48 dist-test-slave-6tjh krb5kdc[22766](info): commencing operation
krb5kdc: starting...
W20260929 02:51:49.121246 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.011s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:49 dist-test-slave-6tjh krb5kdc[22766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:49.397856 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41797
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41797,127.0.97.61:42325,127.0.97.60:46133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:49.738554 22779 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:51:49.738893 22779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:49.738955 22779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:49.739028 22779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:49.746644 22779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:49.746757 22779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:49.746814 22779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:49.746857 22779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:49.746917 22779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:49.757071 22779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41797,127.0.97.61:42325,127.0.97.60:46133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:49.758351 22779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:49.759619 22779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:49.771320 22787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:49.771421 22784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:49.771399 22785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:49.772339 22779 server_base.cc:1034] running on GCE node
I20260929 02:51:49.773217 22779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:49.774654 22779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:49.775898 22779 hybrid_clock.cc:648] HybridClock initialized: now 1790650309775826 us; error 55 us; skew 500 ppm
I20260929 02:51:49.776228 22779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:49 dist-test-slave-6tjh krb5kdc[22766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:49.779661 22779 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:49.781324 22779 webserver.cc:460] Webserver started at http://127.0.97.62:43143/ using document root <none> and password file <none>
I20260929 02:51:49.782058 22779 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:49.782187 22779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:49.782509 22779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:49.785738 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "ead0630a24f749eb86ea3c15597b3da7"
format_stamp: "Formatted at 2026-09-29 02:51:49 on dist-test-slave-6tjh"
server_key: "bdcf0a9bc8e2551b5a5229f04aa96b31"
server_key_iv: "b164542465bc86cd819075255bbae116"
server_key_version: "encryptionkey@0"
I20260929 02:51:49.786554 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "ead0630a24f749eb86ea3c15597b3da7"
format_stamp: "Formatted at 2026-09-29 02:51:49 on dist-test-slave-6tjh"
server_key: "bdcf0a9bc8e2551b5a5229f04aa96b31"
server_key_iv: "b164542465bc86cd819075255bbae116"
server_key_version: "encryptionkey@0"
I20260929 02:51:49.791960 22779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:51:49.795815 22794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:49.797314 22779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:49.797513 22779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "ead0630a24f749eb86ea3c15597b3da7"
format_stamp: "Formatted at 2026-09-29 02:51:49 on dist-test-slave-6tjh"
server_key: "bdcf0a9bc8e2551b5a5229f04aa96b31"
server_key_iv: "b164542465bc86cd819075255bbae116"
server_key_version: "encryptionkey@0"
I20260929 02:51:49.797761 22779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:49.825093 22779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:49.829731 22779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:49.830018 22779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:49.849563 22779 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41797
I20260929 02:51:49.849611 22845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41797 every 8 connection(s)
I20260929 02:51:49.851043 22779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:49.854131 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22779
I20260929 02:51:49.854367 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:49.855223 388 external_mini_cluster.cc:1428] Setting key 97e520b1e2c87f31707803da6083411b
I20260929 02:51:49.855952 22846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:49.866853 22846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 29 02:51:49 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:49.897773 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:49.870440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41797 (local address 127.0.0.1:56382)
0929 02:51:49.871666 (+ 1226us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:49.871713 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:49.874573 (+ 2860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:49.875641 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:49.875652 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:49.876534 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:49.877105 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:49.877137 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:49.878946 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:49.878955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:49.881060 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:49.881083 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:49.882250 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:49.882260 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:49.882475 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:49.883480 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:49.883512 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:49.889481 (+ 5969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:49.893083 (+ 3602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:49.893100 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:49.893118 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:49.893820 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:49.894272 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:49.894281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:49.894287 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:49.894520 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:49.895344 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:49.895356 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:49.895671 (+ 315us) client_negotiation.cc:769] Sending connection context
0929 02:51:49.895972 (+ 301us) client_negotiation.cc:240] Negotiation successful
0929 02:51:49.896681 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":59,"spinlock_wait_cycles":38656,"thread_start_us":716,"threads_started":1}
I20260929 02:51:49.897826 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:49.870440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56382 (local address 127.0.97.62:41797)
0929 02:51:49.871406 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:49.871420 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:49.871866 (+ 446us) server_negotiation.cc:407] Connection header received
0929 02:51:49.875073 (+ 3207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:49.875080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:49.875362 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:49.875541 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:49.877399 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:49.878759 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:49.881267 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:49.881941 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:49.882163 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:49.889819 (+ 7656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:49.889828 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:49.889843 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:49.889880 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:49.892910 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:49.893970 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:49.893977 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:49.893980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:49.894155 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:49.894637 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:49.894641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:49.894643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:49.895192 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:49.895352 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:49.896087 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 02:51:49.896416 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":691,"thread_start_us":387,"threads_started":1}
I20260929 02:51:49.901317 22846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:51:49.902661 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:49.902190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50546)
0929 02:51:49.902414 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:49.902530 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:51:49.902997 22797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
I20260929 02:51:49.904713 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:49.904269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50550)
0929 02:51:49.904415 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:49.904501 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260929 02:51:49.904929 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42325: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 02:51:49.924490 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42325
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41797,127.0.97.61:42325,127.0.97.60:46133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:51:49.954253 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } attempt: 1
I20260929 02:51:49.955487 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:49.954932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50560)
0929 02:51:49.955189 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:49.955321 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 02:51:49.956609 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:49.956154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50576)
0929 02:51:49.956297 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:49.956385 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:51:49.956791 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42325: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
I20260929 02:51:50.025074 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } attempt: 2
I20260929 02:51:50.026433 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.025878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50580)
0929 02:51:50.026117 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.026255 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:51:50.027541 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.027136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50582)
0929 02:51:50.027298 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.027381 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 02:51:50.027796 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42325: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
I20260929 02:51:50.119113 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } attempt: 3
I20260929 02:51:50.120498 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.119846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50590)
0929 02:51:50.120131 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.120279 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260929 02:51:50.121819 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.121306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50598)
0929 02:51:50.121496 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.121605 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:51:50.122092 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42325: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
W20260929 02:51:50.263043 22853 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:51:50.263464 22853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:50.263597 22853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:50.263736 22853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:51:50.265352 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } attempt: 4
I20260929 02:51:50.266664 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.265989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50602)
0929 02:51:50.266365 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.266486 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260929 02:51:50.267817 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.267398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50612)
0929 02:51:50.267558 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.267664 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:51:50.268021 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42325: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42325: connect: Connection refused (error 111)
W20260929 02:51:50.272004 22853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:50.272099 22853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:50.272158 22853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:50.272209 22853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:50.272266 22853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:50.282547 22853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41797,127.0.97.61:42325,127.0.97.60:46133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:50.283913 22853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:50.285441 22853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:50.298614 22858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:50.298686 22861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:50.298683 22859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:50.302501 22853 server_base.cc:1034] running on GCE node
I20260929 02:51:50.303499 22853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:50.305166 22853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:50.306430 22853 hybrid_clock.cc:648] HybridClock initialized: now 1790650310306355 us; error 57 us; skew 500 ppm
I20260929 02:51:50.306802 22853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:50.314881 22853 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 02:51:50.316731 22853 webserver.cc:460] Webserver started at http://127.0.97.61:38869/ using document root <none> and password file <none>
I20260929 02:51:50.317520 22853 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:50.317657 22853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:50.318001 22853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:50.321354 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data/instance:
uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1"
format_stamp: "Formatted at 2026-09-29 02:51:50 on dist-test-slave-6tjh"
server_key: "87652aac7f8307338d405ed7ea269226"
server_key_iv: "fcc6022c1281e2f69209a8d2be3b2111"
server_key_version: "encryptionkey@0"
I20260929 02:51:50.322330 22853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/wal/instance:
uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1"
format_stamp: "Formatted at 2026-09-29 02:51:50 on dist-test-slave-6tjh"
server_key: "87652aac7f8307338d405ed7ea269226"
server_key_iv: "fcc6022c1281e2f69209a8d2be3b2111"
server_key_version: "encryptionkey@0"
I20260929 02:51:50.328367 22853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 02:51:50.332325 22868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:50.334048 22853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:50.334273 22853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/wal
uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1"
format_stamp: "Formatted at 2026-09-29 02:51:50 on dist-test-slave-6tjh"
server_key: "87652aac7f8307338d405ed7ea269226"
server_key_iv: "fcc6022c1281e2f69209a8d2be3b2111"
server_key_version: "encryptionkey@0"
I20260929 02:51:50.334498 22853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:50.347626 22853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:50.352248 22853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:50.352581 22853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:50.371897 22853 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42325
I20260929 02:51:50.371922 22919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42325 every 8 connection(s)
I20260929 02:51:50.373404 22853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
I20260929 02:51:50.377362 22920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:50.380988 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22853
I20260929 02:51:50.381186 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-1/wal/instance
I20260929 02:51:50.381743 388 external_mini_cluster.cc:1428] Setting key ad4f008655a92d19a76a74fdc00cb80c
I20260929 02:51:50.386067 22920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 02:51:50.411343 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.389655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.0.97.62:41797)
0929 02:51:50.389916 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.389925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.391150 (+ 1225us) server_negotiation.cc:407] Connection header received
0929 02:51:50.393554 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.393560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.393634 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.393738 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.395994 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.396810 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.399359 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.399800 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.399907 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.404468 (+ 4561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.404473 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.404476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.404514 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.407456 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.408322 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.408334 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.408337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.408475 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.408975 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.408983 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.408987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.409575 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.409780 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.410667 (+ 887us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.410950 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 02:51:50.412917 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.390077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41797 (local address 127.0.0.1:56390)
0929 02:51:50.390958 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.391006 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:50.393220 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:50.394013 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:50.394028 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:50.394851 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:50.395744 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.395774 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.396993 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.397001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.399162 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.399182 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.400045 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.400054 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.400421 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.401265 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:50.401303 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:50.404193 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:50.407635 (+ 3442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.407646 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.407666 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.408178 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.408629 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.408634 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.408638 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.408844 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.409755 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:50.409765 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:50.410224 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 02:51:50.410565 (+ 341us) client_negotiation.cc:240] Negotiation successful
0929 02:51:50.411278 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":419,"threads_started":1}
I20260929 02:51:50.414932 22920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:51:50.431892 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.415992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50628)
0929 02:51:50.416282 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.416305 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:50.416435 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:50.417422 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:50.417430 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:50.418009 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:50.418248 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.418263 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.420149 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.420158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.421686 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.421699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.422622 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.422629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.422809 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.423453 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:50.423472 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:50.425722 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:50.429257 (+ 3535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.429265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.429271 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.429657 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.430044 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.430052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.430056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.430205 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.430829 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:50.430840 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:50.430987 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 02:51:50.431142 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 02:51:50.431457 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260929 02:51:50.432029 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.416164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.0.97.61:42325)
0929 02:51:50.416939 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.416945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.416972 (+ 27us) server_negotiation.cc:407] Connection header received
0929 02:51:50.417037 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.417043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.417240 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.417370 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.418453 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.419965 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.421834 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.422442 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.422605 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.425944 (+ 3339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.425956 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.425972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.426002 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.429102 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.429799 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.429808 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.429812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.429929 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.430310 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.430319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.430322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.430683 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.430911 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.431299 (+ 388us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.431594 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":623,"thread_start_us":477,"threads_started":1}
I20260929 02:51:50.435053 22920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:51:50.436246 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.435754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56398)
0929 02:51:50.435947 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.436087 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:51:50.436508 22870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.438282 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.437869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56408)
0929 02:51:50.438084 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.438167 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 02:51:50.438524 22920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 02:51:50.447438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46133
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41797,127.0.97.61:42325,127.0.97.60:46133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:51:50.487859 22920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 1
I20260929 02:51:50.489075 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.488625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56424)
0929 02:51:50.488811 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.488913 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 02:51:50.490196 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.489830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56430)
0929 02:51:50.489980 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.490074 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:51:50.490382 22920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.558609 22920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 2
I20260929 02:51:50.559891 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.559418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56436)
0929 02:51:50.559625 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.559729 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 02:51:50.561203 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.560816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56448)
0929 02:51:50.560974 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.561048 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:51:50.561403 22920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.567269 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } attempt: 5
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:51:50.583261 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.568095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50638)
0929 02:51:50.568303 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.568327 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:50.568453 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:50.568889 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:50.568893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:50.569332 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:50.569550 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.569560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.571091 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.571096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.572005 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.572020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.573186 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.573190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.573343 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.573969 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:50.573989 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:50.576023 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:50.580134 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.580140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.580143 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.580651 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.581119 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.581122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.581125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.581254 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.582080 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:50.582089 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:50.582334 (+ 245us) client_negotiation.cc:769] Sending connection context
0929 02:51:50.582525 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 02:51:50.582845 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260929 02:51:50.583261 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.568107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50638 (local address 127.0.97.61:42325)
0929 02:51:50.568314 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.568321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.568412 (+ 91us) server_negotiation.cc:407] Connection header received
0929 02:51:50.568617 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.568623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.568715 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.568859 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.569739 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.570890 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.572182 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.573027 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.573177 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.576246 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.576254 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.576259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.576304 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.579971 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.580796 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.580802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.580806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.580968 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.581377 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.581385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.581391 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.581877 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.582060 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.582644 (+ 584us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.582881 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:51:50.584672 22846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:51:50.585904 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.585487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56454)
0929 02:51:50.585664 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.585746 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 02:51:50.587008 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.586625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56456)
0929 02:51:50.586810 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.586895 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 02:51:50.587288 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.638562 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 1
I20260929 02:51:50.639968 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.639429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56464)
0929 02:51:50.639680 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.639790 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260929 02:51:50.641085 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.640694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56468)
0929 02:51:50.640863 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.640943 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 02:51:50.641402 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.652665 22920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 3
I20260929 02:51:50.653891 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.653419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56476)
0929 02:51:50.653640 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.653733 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 02:51:50.655171 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.654665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56482)
0929 02:51:50.654939 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.655030 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260929 02:51:50.655380 22920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.709642 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 2
I20260929 02:51:50.710994 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.710424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56490)
0929 02:51:50.710650 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.710775 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 02:51:50.712159 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.711774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56500)
0929 02:51:50.711955 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.712026 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 02:51:50.712426 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.798633 22920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 4
W20260929 02:51:50.799211 22927 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:51:50.799697 22927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:50.799803 22927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:50.799945 22927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 02:51:50.799962 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.799378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56510)
0929 02:51:50.799656 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.799767 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 02:51:50.801018 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.800686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56514)
0929 02:51:50.800830 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.800908 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 02:51:50.801349 22920 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
W20260929 02:51:50.808269 22927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:50.808413 22927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:50.808516 22927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:50.808629 22927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:50.808724 22927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:50.818672 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 3
I20260929 02:51:50.819245 22927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41797,127.0.97.61:42325,127.0.97.60:46133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:50.820001 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.819408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56522)
0929 02:51:50.819740 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.819831 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260929 02:51:50.820782 22927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:50.821179 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.820768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56530)
0929 02:51:50.820978 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.821066 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260929 02:51:50.821393 22846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46133: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46133: connect: Connection refused (error 111)
I20260929 02:51:50.822245 22927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:50.834915 22935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:50.834949 22932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:50.835072 22933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:50.835896 22927 server_base.cc:1034] running on GCE node
I20260929 02:51:50.836825 22927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:50.838465 22927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:50.839720 22927 hybrid_clock.cc:648] HybridClock initialized: now 1790650310839654 us; error 40 us; skew 500 ppm
I20260929 02:51:50.840123 22927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:50.843591 22927 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 02:51:50.845264 22927 webserver.cc:460] Webserver started at http://127.0.97.60:33447/ using document root <none> and password file <none>
I20260929 02:51:50.845976 22927 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:50.846103 22927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:50.846429 22927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:50.849560 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data/instance:
uuid: "4ac9034486814a9397358cd54306b15b"
format_stamp: "Formatted at 2026-09-29 02:51:50 on dist-test-slave-6tjh"
server_key: "f0b372052a133c4ead0dcac9398fe34f"
server_key_iv: "53f8616e847697a3a15e46f58a7c4b29"
server_key_version: "encryptionkey@0"
I20260929 02:51:50.850373 22927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/wal/instance:
uuid: "4ac9034486814a9397358cd54306b15b"
format_stamp: "Formatted at 2026-09-29 02:51:50 on dist-test-slave-6tjh"
server_key: "f0b372052a133c4ead0dcac9398fe34f"
server_key_iv: "53f8616e847697a3a15e46f58a7c4b29"
server_key_version: "encryptionkey@0"
I20260929 02:51:50.855798 22927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:51:50.859459 22942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:50.861011 22927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:51:50.861223 22927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/wal
uuid: "4ac9034486814a9397358cd54306b15b"
format_stamp: "Formatted at 2026-09-29 02:51:50 on dist-test-slave-6tjh"
server_key: "f0b372052a133c4ead0dcac9398fe34f"
server_key_iv: "53f8616e847697a3a15e46f58a7c4b29"
server_key_version: "encryptionkey@0"
I20260929 02:51:50.861449 22927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:50.893589 22927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:50.898030 22927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:50.898330 22927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:50.918224 22927 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46133
I20260929 02:51:50.918258 22993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46133 every 8 connection(s)
I20260929 02:51:50.919625 22927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
I20260929 02:51:50.923416 22994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:50.925184 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 22927
I20260929 02:51:50.925372 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790649949495923-388-0/minicluster-data/master-2/wal/instance
I20260929 02:51:50.925936 388 external_mini_cluster.cc:1428] Setting key da99582f003916648727e0e313a5c965
I20260929 02:51:50.933633 22994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:50.950531 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.927821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56538 (local address 127.0.97.60:46133)
0929 02:51:50.928680 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.928694 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.928740 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:51:50.930309 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.930337 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.930753 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.931091 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.933337 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.934806 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.936697 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.937808 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.938044 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.942484 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.942494 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.942507 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.942545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.945887 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.946640 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.946646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.946653 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.946859 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.947252 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.947259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.947264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.948047 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.948359 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.948784 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.949250 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":390,"threads_started":1}
I20260929 02:51:50.951781 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.936695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41797 (local address 127.0.0.1:56396)
0929 02:51:50.937716 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.937764 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:50.937919 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:50.938380 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:50.938388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:50.939094 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:50.939386 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.939401 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.940507 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.940512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.941673 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.941685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.942474 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.942481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.942650 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.943337 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:50.943358 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:50.945802 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:50.949374 (+ 3572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.949383 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.949387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.949818 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.950216 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.950219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.950221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.950333 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.950961 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:50.950967 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:50.951122 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 02:51:50.951264 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 02:51:50.951502 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":608,"threads_started":1}
I20260929 02:51:50.952296 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.936698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.0.97.62:41797)
0929 02:51:50.937330 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.937337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.937864 (+ 527us) server_negotiation.cc:407] Connection header received
0929 02:51:50.938088 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.938092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.938154 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.938293 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.939536 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.940358 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.941823 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.942229 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.942364 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.946010 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.946017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.946022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.946061 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.949254 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.949957 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.949962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.949965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.950113 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.950445 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.950452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.950458 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.950835 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.950997 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.951412 (+ 415us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.951629 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":287,"threads_started":1}
I20260929 02:51:50.953683 22994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:50.970072 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.954500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42325 (local address 127.0.0.1:50640)
0929 02:51:50.954668 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.954687 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:50.954796 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:50.955110 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:50.955117 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:50.955586 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:50.955758 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.955770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.957377 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.957382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.958831 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.958847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.959842 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.959847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.960029 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.960806 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:50.960833 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:50.963487 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:50.967215 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.967220 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.967224 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.967645 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.968128 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.968132 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.968135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.968290 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.968969 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:50.968973 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:50.969184 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 02:51:50.969358 (+ 174us) client_negotiation.cc:240] Negotiation successful
0929 02:51:50.969705 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 02:51:50.970095 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.954504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.0.97.61:42325)
0929 02:51:50.954682 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.954688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.954734 (+ 46us) server_negotiation.cc:407] Connection header received
0929 02:51:50.954923 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.954927 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.954984 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.955055 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.955943 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.957210 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.959001 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.959677 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.959806 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.963742 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.963747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.963751 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.963787 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.967074 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.967771 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.967777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.967780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.967973 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.968416 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.968424 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.968429 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.968838 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.968988 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.969430 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.969664 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 02:51:50.971210 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.953804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.0.97.62:41797)
0929 02:51:50.954013 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.954018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.954036 (+ 18us) server_negotiation.cc:407] Connection header received
0929 02:51:50.954152 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.954156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.954262 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.954362 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.955940 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.957245 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.959001 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.959677 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.959806 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.964571 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.964581 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.964586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.964628 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.967744 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.968482 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.968487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.968489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.968694 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.969091 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.969096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.969099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.969479 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.969784 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.970097 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.970888 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 02:51:50.971104 22994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 02:51:50 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:51:50.986174 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.972032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56540)
0929 02:51:50.972254 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.972275 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:50.972391 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:50.972728 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:50.972733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:50.973178 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:50.973378 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.973390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.974929 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.974933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.976237 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:50.976251 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.977326 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.977330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:50.977491 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.978139 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:50.978157 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:50.980474 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:50.983732 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.983738 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.983742 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.984188 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.984526 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:50.984532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:50.984536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:50.984690 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:50.985174 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:50.985178 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:50.985311 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 02:51:50.985448 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 02:51:50.985759 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260929 02:51:50.986331 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.972289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.0.97.60:46133)
0929 02:51:50.972429 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.972434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.972448 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:51:50.972488 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.972491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.972588 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.972680 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.973501 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.974776 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.976401 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.977157 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.977304 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.980706 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.980715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.980722 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.980767 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.983590 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.984303 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.984309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.984312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.984427 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.984788 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.984795 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.984798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.985069 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.985211 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.985536 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.985994 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":210,"server-negotiator.queue_time_us":40}
I20260929 02:51:50.989657 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.973072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50642 (local address 127.0.97.61:42325)
0929 02:51:50.973238 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.973243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.973264 (+ 21us) server_negotiation.cc:407] Connection header received
0929 02:51:50.973413 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.973420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.973501 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.973587 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:50.975237 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.976311 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.978049 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:50.978454 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:50.978550 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:50.983513 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:50.983518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:50.983522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:50.983563 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:50.986824 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.987582 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.987586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.987588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.987735 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:50.988195 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:50.988198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:50.988201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:50.988463 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:50.988794 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:50.989087 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 02:51:50.989347 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 02:51:50.991549 22994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: Bootstrap starting.
I20260929 02:51:50.994299 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:50.995100 22994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:50.996112 22994 log.cc:824] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:50.998610 22846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 4
I20260929 02:51:50.999277 22994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: No bootstrap required, opened a new log
I20260929 02:51:50.996955 22821 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ead0630a24f749eb86ea3c15597b3da7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56410
Sep 29 02:51:51 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 02:51:51.010056 22994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.010447 22994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ac9034486814a9397358cd54306b15b, State: Initialized, Role: FOLLOWER
I20260929 02:51:51.011049 22994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.011978 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.012533 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:51.013598 22994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:51.014390 22850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.999207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56556)
0929 02:51:50.999431 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:50.999453 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:50.999558 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:51.000015 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:51.000020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:51.000484 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:51.000718 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:51.000727 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.001994 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.002003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:51.003066 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:51.003078 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.003883 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.003889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:51.004109 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.004758 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:51.004776 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:51.007282 (+ 2506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:51.011413 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:51.011419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:51.011423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:51.011998 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:51.012509 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:51.012515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:51.012521 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:51.012744 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:51.013315 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:51.013322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:51.013505 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 02:51:51.013659 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 02:51:51.013986 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260929 02:51:51.014691 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:50.999226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56556 (local address 127.0.97.60:46133)
0929 02:51:50.999763 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:50.999768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:50.999781 (+ 13us) server_negotiation.cc:407] Connection header received
0929 02:51:50.999823 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:50.999827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:50.999878 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:50.999986 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.000866 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.001824 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.003213 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.003712 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.003870 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.007489 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:51.007495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:51.007502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:51.007544 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:51.011242 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.012141 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.012148 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.012151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.012335 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.012849 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.012856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.012859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.013189 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:51.013380 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.013846 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.014070 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420}
I20260929 02:51:51.019616 22846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7: Bootstrap starting.
I20260929 02:51:51.024005 22846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:51.025470 22846 log.cc:824] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7: Log is configured to *not* fsync() on all Append() calls
W20260929 02:51:51.028810 23012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:51:51.028951 23012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:51:51.035784 22846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7: No bootstrap required, opened a new log
I20260929 02:51:51.026679 22927 master_runner.cc:186] Error getting master registration for 127.0.97.62:41797: OK, instance_id { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" instance_seqno: 1790650309843945 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:51:51.039943 22927 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:51:51.042740 22846 raft_consensus.cc:348] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.043130 22846 raft_consensus.cc:729] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ead0630a24f749eb86ea3c15597b3da7, State: Initialized, Role: FOLLOWER
I20260929 02:51:51.043607 22846 consensus_queue.cc:260] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.044956 23015 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.045634 23015 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:51.046299 22846 sys_catalog.cc:564] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:51.046425 22821 raft_consensus.cc:482] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:51:51.046607 22821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:51.048878 22821 raft_consensus.cc:504] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.051589 22821 leader_election.cc:288] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [CANDIDATE]: Term 1 election: Requested vote from peers 35c7972ffbdb41b3bc27cdaa754a0ad1 (127.0.97.61:42325), 4ac9034486814a9397358cd54306b15b (127.0.97.60:46133)
I20260929 02:51:51.051915 22969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ead0630a24f749eb86ea3c15597b3da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4ac9034486814a9397358cd54306b15b"
I20260929 02:51:51.052461 22969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:51.056141 22969 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ead0630a24f749eb86ea3c15597b3da7 in term 1.
I20260929 02:51:51.056850 22796 leader_election.cc:302] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ac9034486814a9397358cd54306b15b, ead0630a24f749eb86ea3c15597b3da7; no voters:
I20260929 02:51:51.057404 23015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:51.064921 23015 raft_consensus.cc:686] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [term 1 LEADER]: Becoming Leader. State: Replica: ead0630a24f749eb86ea3c15597b3da7, State: Running, Role: LEADER
I20260929 02:51:51.066191 23015 consensus_queue.cc:237] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.076393 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ead0630a24f749eb86ea3c15597b3da7. Latest consensus state: current_term: 1 leader_uuid: "ead0630a24f749eb86ea3c15597b3da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.076912 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: This master's current role is: LEADER
W20260929 02:51:51.082988 22800 tablet.cc:2369] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:51.095187 23026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 02:51:51.100582 22920 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } attempt: 5
Sep 29 02:51:51 dist-test-slave-6tjh krb5kdc[22766](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 02:51:51.120196 23026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:51.133873 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.083947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56566 (local address 127.0.97.60:46133)
0929 02:51:51.084181 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.084188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.085913 (+ 1725us) server_negotiation.cc:407] Connection header received
0929 02:51:51.086521 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.086530 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.086641 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.086738 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.090808 (+ 4070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.091945 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.098518 (+ 6573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.099545 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.099688 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.120741 (+ 21053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:51.120748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:51.120755 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:51.120800 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:51.125763 (+ 4963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.126933 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.126943 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.126949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.127174 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.130453 (+ 3279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.130462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.130468 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.130972 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:51.131257 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.132941 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.133289 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 02:51:51.141271 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.087202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56414 (local address 127.0.97.62:41797)
0929 02:51:51.087447 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.087456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.092514 (+ 5058us) server_negotiation.cc:407] Connection header received
0929 02:51:51.094043 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.094052 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.094160 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.094272 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.099056 (+ 4784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.100328 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.111064 (+ 10736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.111806 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.111921 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.128883 (+ 16962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:51.128893 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:51.128903 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:51.128952 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:51.133307 (+ 4355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.134804 (+ 1497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.134835 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.134839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.135120 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.136461 (+ 1341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.136469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.136497 (+ 28us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.137133 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:51.137587 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.139896 (+ 2309us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.140637 (+ 741us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:51:51.147115 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.085279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.0.97.61:42325)
0929 02:51:51.085515 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.085524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.091638 (+ 6114us) server_negotiation.cc:407] Connection header received
0929 02:51:51.092897 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.092906 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.093019 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.093107 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.105336 (+ 12229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.106781 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.118820 (+ 12039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.119467 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.119580 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.137659 (+ 18079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:51.137668 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:51.137673 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:51.137732 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:51.142716 (+ 4984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.143891 (+ 1175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.143901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.143907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.144103 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.144684 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.144703 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.144710 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.145184 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:51.145508 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.146206 (+ 698us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.146571 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:51:51.149932 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.101196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46133 (local address 127.0.0.1:56582)
0929 02:51:51.101372 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:51.101391 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:51.101521 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:51.109069 (+ 7548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:51.109077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:51.109741 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:51.110036 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:51.110054 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.112269 (+ 2215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.112278 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:51.114382 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:51.114409 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.115861 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.115870 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:51.116157 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.116838 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:51.116869 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:51.120856 (+ 3987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:51.146231 (+ 25375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:51.146237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:51.146251 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:51.146810 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:51.147586 (+ 776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:51.147591 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:51.147596 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:51.147771 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:51.148894 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:51.148900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:51.149062 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 02:51:51.149196 (+ 134us) client_negotiation.cc:240] Negotiation successful
0929 02:51:51.149507 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
W20260929 02:51:51.150521 22948 tablet.cc:2369] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:51.151758 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.101613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.0.97.60:46133)
0929 02:51:51.108743 (+ 7130us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.108748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.108770 (+ 22us) server_negotiation.cc:407] Connection header received
0929 02:51:51.108820 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.108823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.108886 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.109021 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.110312 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.112038 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.114684 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.115622 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.116536 (+ 914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.121272 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:51.121281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:51.121289 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:51.121334 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:51.145985 (+ 24651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.147072 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.147079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.147084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.147395 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:51.147966 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:51.147971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:51.148009 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:51.148687 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:51.150427 (+ 1740us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.150679 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.151111 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6984,"thread_start_us":6700,"threads_started":1}
I20260929 02:51:51.158521 22969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [term 1 FOLLOWER]: Refusing update from remote peer ead0630a24f749eb86ea3c15597b3da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:51.159571 23015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:51.162115 22920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1: Bootstrap starting.
I20260929 02:51:51.170344 22920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:51.172215 22920 log.cc:824] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:51.162854 22779 master_runner.cc:186] Error getting master registration for 127.0.97.61:42325: OK, instance_id { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" instance_seqno: 1790650310366522 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:51:51.178498 22779 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:51:51.182739 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ead0630a24f749eb86ea3c15597b3da7. Latest consensus state: current_term: 1 leader_uuid: "ead0630a24f749eb86ea3c15597b3da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.183033 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:51.184226 22920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1: No bootstrap required, opened a new log
I20260929 02:51:51.186928 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ead0630a24f749eb86ea3c15597b3da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.187923 23001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ead0630a24f749eb86ea3c15597b3da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.188176 23001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:51.195722 22920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.196134 22920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c7972ffbdb41b3bc27cdaa754a0ad1, State: Initialized, Role: FOLLOWER
I20260929 02:51:51.196677 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:51.196666 22920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } }
I20260929 02:51:51.198078 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.198820 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:51.199112 22920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:51.199204 22895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ead0630a24f749eb86ea3c15597b3da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1"
I20260929 02:51:51.199647 23026 catalog_manager.cc:1347] Generated new cluster ID: 988b7cf175f141a7a6141086d3d3076d
I20260929 02:51:51.199647 22895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:51.199755 23026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:51.203081 22895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ead0630a24f749eb86ea3c15597b3da7 in term 1.
I20260929 02:51:51.204008 22894 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [term 1 FOLLOWER]: Refusing update from remote peer ead0630a24f749eb86ea3c15597b3da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:51.205628 23020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:51.235184 23050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ead0630a24f749eb86ea3c15597b3da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.235864 23050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:51.236083 23045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ead0630a24f749eb86ea3c15597b3da7. Latest consensus state: current_term: 1 leader_uuid: "ead0630a24f749eb86ea3c15597b3da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.236411 23045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:51.238564 23020 sys_catalog.cc:455] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ead0630a24f749eb86ea3c15597b3da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ead0630a24f749eb86ea3c15597b3da7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41797 } } peers { permanent_uuid: "35c7972ffbdb41b3bc27cdaa754a0ad1" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42325 } } peers { permanent_uuid: "4ac9034486814a9397358cd54306b15b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46133 } } }
I20260929 02:51:51.238879 23020 sys_catalog.cc:458] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:51.251348 23026 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:51.253957 23026 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:51.277136 23026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ead0630a24f749eb86ea3c15597b3da7: Generated new TSK 0
I20260929 02:51:51.278250 23026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:51.280902 23062 catalog_manager.cc:1259] Loaded cluster ID: 988b7cf175f141a7a6141086d3d3076d
I20260929 02:51:51.281013 23062 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1: loading cluster ID for follower catalog manager: success
I20260929 02:51:51.284911 23062 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1: acquiring CA information for follower catalog manager: success
I20260929 02:51:51.305356 23062 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 35c7972ffbdb41b3bc27cdaa754a0ad1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:51:51.431146 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.411860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.0.97.60:46133)
0929 02:51:51.412059 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.412066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.429245 (+ 17179us) server_negotiation.cc:407] Connection header received
0929 02:51:51.429426 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.429432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.429555 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.429684 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.430901 (+ 1217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56584: BlockingRecv error: recv got EOF from 127.0.0.1:56584 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 02:51:51.431228 22924 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.409491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.0.97.61:42325)
0929 02:51:51.409763 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.409770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.427632 (+ 17862us) server_negotiation.cc:407] Connection header received
0929 02:51:51.429775 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.429781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.429874 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.429945 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.431001 (+ 1056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50656: BlockingRecv error: recv got EOF from 127.0.0.1:50656 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260929 02:51:51.433387 22998 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.424900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.0.97.62:41797)
0929 02:51:51.425137 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.425144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.431692 (+ 6548us) server_negotiation.cc:407] Connection header received
0929 02:51:51.431929 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.431941 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.432024 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.432095 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.433106 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56420: BlockingRecv error: recv got EOF from 127.0.0.1:56420 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 02:51:51.492022 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.475486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.0.97.61:42325)
0929 02:51:51.475691 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.475698 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.483348 (+ 7650us) server_negotiation.cc:407] Connection header received
0929 02:51:51.483925 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.483932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.484207 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.484362 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:51.485105 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.486322 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.488462 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.489187 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.489296 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.490236 (+ 940us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:51.491037 (+ 801us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:51.491170 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.491465 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.491587 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260929 02:51:51.548290 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.534686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.0.97.60:46133)
0929 02:51:51.534880 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.534887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.545732 (+ 10845us) server_negotiation.cc:407] Connection header received
0929 02:51:51.546402 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.546410 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.546574 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.546666 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.547948 (+ 1282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56598: BlockingRecv error: recv got EOF from 127.0.0.1:56598 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 02:51:51.595587 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.579102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.0.97.61:42325)
0929 02:51:51.579335 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.579342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.587319 (+ 7977us) server_negotiation.cc:407] Connection header received
0929 02:51:51.587930 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.587937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.588061 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.588137 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:51.588945 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.590338 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.592723 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.593589 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.593719 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.594477 (+ 758us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:51.594689 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:51.594789 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.595053 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.595185 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 02:51:51.647993 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.634478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.0.97.60:46133)
0929 02:51:51.634707 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.634715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.645708 (+ 10993us) server_negotiation.cc:407] Connection header received
0929 02:51:51.646353 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.646359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.646470 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.646568 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.647748 (+ 1180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56606: BlockingRecv error: recv got EOF from 127.0.0.1:56606 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 02:51:51.709622 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.690755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50678 (local address 127.0.97.61:42325)
0929 02:51:51.690971 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.690978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.702400 (+ 11422us) server_negotiation.cc:407] Connection header received
0929 02:51:51.702737 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.702744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.702866 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.702940 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:51.703705 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.704911 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.707046 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.707775 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.707884 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.708607 (+ 723us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:51.708764 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:51.708845 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.709144 (+ 299us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.709249 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260929 02:51:51.779356 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.764849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.0.97.60:46133)
0929 02:51:51.765060 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.765073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.776967 (+ 11894us) server_negotiation.cc:407] Connection header received
0929 02:51:51.777601 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.777611 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.777738 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.777845 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.779084 (+ 1239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56614: BlockingRecv error: recv got EOF from 127.0.0.1:56614 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:51:51.852394 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.842376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50684 (local address 127.0.97.61:42325)
0929 02:51:51.842584 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.842591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.845748 (+ 3157us) server_negotiation.cc:407] Connection header received
0929 02:51:51.846024 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.846031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.846130 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.846259 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:51.846794 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.847734 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.849627 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.850401 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.850629 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.851411 (+ 782us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:51.851574 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:51.851653 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.851931 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.852030 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 02:51:51.879864 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.877573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.0.97.60:46133)
0929 02:51:51.877850 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.877856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.878199 (+ 343us) server_negotiation.cc:407] Connection header received
0929 02:51:51.878408 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.878414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.878532 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.878658 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.879622 (+ 964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56618: BlockingRecv error: recv got EOF from 127.0.0.1:56618 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
I20260929 02:51:51.897210 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.891547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.0.97.61:42325)
0929 02:51:51.891759 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.891765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.892164 (+ 399us) server_negotiation.cc:407] Connection header received
0929 02:51:51.892414 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.892419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.892493 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.892638 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:51.893073 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.893845 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.895193 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.895656 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.895799 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.896232 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:51.896342 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:51.896454 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.896767 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.896857 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 02:51:51.906373 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.904332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.0.97.60:46133)
0929 02:51:51.904529 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.904537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.904917 (+ 380us) server_negotiation.cc:407] Connection header received
0929 02:51:51.905081 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.905087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.905190 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.905321 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.906110 (+ 789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56632: BlockingRecv error: recv got EOF from 127.0.0.1:56632 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:51:51.932148 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.925969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50708 (local address 127.0.97.61:42325)
0929 02:51:51.926191 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.926198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.926550 (+ 352us) server_negotiation.cc:407] Connection header received
0929 02:51:51.926724 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.926730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.926824 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.926936 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:51.927385 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.928620 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.930183 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.930850 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.930977 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.931411 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:51.931552 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:51.931644 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.931772 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.931865 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 02:51:51.940165 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.938100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56648 (local address 127.0.97.60:46133)
0929 02:51:51.938350 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.938356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.938765 (+ 409us) server_negotiation.cc:407] Connection header received
0929 02:51:51.938947 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.938950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.939022 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.939108 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.939941 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56648: BlockingRecv error: recv got EOF from 127.0.0.1:56648 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 02:51:51.981223 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:51.975723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50712 (local address 127.0.97.61:42325)
0929 02:51:51.975891 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.975896 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.976260 (+ 364us) server_negotiation.cc:407] Connection header received
0929 02:51:51.976519 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.976523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.976620 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.976716 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:51.977150 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.977939 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.979253 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:51.979801 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:51.979933 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:51.980407 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:51.980614 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:51.980716 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:51.980864 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 02:51:51.980945 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 02:51:51.989279 23033 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:51.986921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.0.97.60:46133)
0929 02:51:51.987223 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:51.987230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:51.987640 (+ 410us) server_negotiation.cc:407] Connection header received
0929 02:51:51.987876 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:51.987881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:51.987988 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:51.988117 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:51.989002 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56662: BlockingRecv error: recv got EOF from 127.0.0.1:56662 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
I20260929 02:51:52.030956 23012 catalog_manager.cc:1259] Loaded cluster ID: 988b7cf175f141a7a6141086d3d3076d
I20260929 02:51:52.031118 23012 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: loading cluster ID for follower catalog manager: success
I20260929 02:51:52.033794 23012 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: acquiring CA information for follower catalog manager: success
I20260929 02:51:52.035548 23012 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4ac9034486814a9397358cd54306b15b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:51:52.062047 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:52.057079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.0.97.61:42325)
0929 02:51:52.057326 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:52.057332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:52.057609 (+ 277us) server_negotiation.cc:407] Connection header received
0929 02:51:52.057772 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:52.057776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:52.057847 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:52.057928 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:52.058334 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.059063 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.060368 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.060875 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.060980 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:52.061363 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:52.061461 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:52.061547 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:52.061668 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 02:51:52.061731 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:51:52.073760 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:52.067988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.0.97.60:46133)
0929 02:51:52.068247 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:52.068254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:52.068412 (+ 158us) server_negotiation.cc:407] Connection header received
0929 02:51:52.068630 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:52.068635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:52.068790 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:52.068901 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:52.069371 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.070128 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.071432 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.071856 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.071956 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:52.072481 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:52.073118 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:52.073246 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:52.073347 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 02:51:52.073421 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 02:51:52.090003 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:52.080618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.97.61:42325)
0929 02:51:52.080862 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:52.080870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:52.082775 (+ 1905us) server_negotiation.cc:407] Connection header received
0929 02:51:52.083021 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:52.083026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:52.083111 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:52.083169 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:52.084152 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.085066 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.087422 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.088211 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.088344 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:52.089230 (+ 886us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:52.089389 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:52.089470 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:52.089606 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 02:51:52.089688 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 02:51:52.090050 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:52.080172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56676 (local address 127.0.97.60:46133)
0929 02:51:52.080377 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:52.080383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:52.082502 (+ 2119us) server_negotiation.cc:407] Connection header received
0929 02:51:52.082936 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:52.082944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:52.083034 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:52.083097 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:52.084645 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.085855 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.087944 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.088450 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.088610 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:52.089130 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:52.089280 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:52.089373 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:52.089602 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 02:51:52.089699 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:51:52.092278 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:52.080214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.0.97.62:41797)
0929 02:51:52.084640 (+ 4426us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:52.084647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:52.084667 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:51:52.084794 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:52.084801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:52.085041 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:52.085559 (+ 518us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:52.085892 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.086779 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.089227 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:52.089990 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:52.090142 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:52.090882 (+ 740us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:52.091335 (+ 453us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:52.091472 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:52.091876 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 02:51:52.091968 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4239,"thread_start_us":4131,"threads_started":1}
I20260929 02:51:52.096912 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22779
I20260929 02:51:52.098606 22841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:52.211346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22779
W20260929 02:51:52.232002 22946 connection.cc:535] server connection from 127.0.0.1:56556 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:51:52.232723 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22853
I20260929 02:51:52.233788 22915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:52.343992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22853
I20260929 02:51:52.364086 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 22927
I20260929 02:51:52.365303 22989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:52.474246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 22927
2026-09-29T02:51:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4476 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:51:52 dist-test-slave-6tjh krb5kdc[23166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:51:52 dist-test-slave-6tjh krb5kdc[23166](info): set up 2 sockets
Sep 29 02:51:52 dist-test-slave-6tjh krb5kdc[23166](info): commencing operation
krb5kdc: starting...
W20260929 02:51:53.609287 388 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/test-user.keytab.
Sep 29 02:51:53 dist-test-slave-6tjh krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260929 02:51:53.892608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38433
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38433,127.0.97.61:36195,127.0.97.60:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:51:54.231802 23179 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:51:54.232142 23179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:54.232270 23179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:54.232398 23179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:54.240001 23179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:54.240119 23179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:54.240176 23179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:54.240278 23179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:54.240381 23179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:54.250862 23179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38433,127.0.97.61:36195,127.0.97.60:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:54.252251 23179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:54.253664 23179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:54.266023 23184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:54.265980 23185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:54.266027 23187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:54.266721 23179 server_base.cc:1034] running on GCE node
I20260929 02:51:54.267599 23179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:54.269196 23179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:54.270476 23179 hybrid_clock.cc:648] HybridClock initialized: now 1790650314270389 us; error 59 us; skew 500 ppm
I20260929 02:51:54.270844 23179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:54 dist-test-slave-6tjh krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:54.274335 23179 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260929 02:51:54.275907 23179 webserver.cc:460] Webserver started at http://127.0.97.62:42139/ using document root <none> and password file <none>
I20260929 02:51:54.276731 23179 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:54.276870 23179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:54.277213 23179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:54.280293 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data/instance:
uuid: "4c063aef7c6041a495e2bb1a3da85404"
format_stamp: "Formatted at 2026-09-29 02:51:54 on dist-test-slave-6tjh"
server_key: "ce19ca3ae03b699bf55a6adf6a6a07ba"
server_key_iv: "950fe697c9dd03bd646f574652f42556"
server_key_version: "encryptionkey@0"
I20260929 02:51:54.281174 23179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/wal/instance:
uuid: "4c063aef7c6041a495e2bb1a3da85404"
format_stamp: "Formatted at 2026-09-29 02:51:54 on dist-test-slave-6tjh"
server_key: "ce19ca3ae03b699bf55a6adf6a6a07ba"
server_key_iv: "950fe697c9dd03bd646f574652f42556"
server_key_version: "encryptionkey@0"
I20260929 02:51:54.286631 23179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:51:54.290449 23194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:54.291903 23179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:51:54.292157 23179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/wal
uuid: "4c063aef7c6041a495e2bb1a3da85404"
format_stamp: "Formatted at 2026-09-29 02:51:54 on dist-test-slave-6tjh"
server_key: "ce19ca3ae03b699bf55a6adf6a6a07ba"
server_key_iv: "950fe697c9dd03bd646f574652f42556"
server_key_version: "encryptionkey@0"
I20260929 02:51:54.292438 23179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:54.304526 23179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:54.308804 23179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:54.309098 23179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:54.329205 23179 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38433
I20260929 02:51:54.329228 23245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38433 every 8 connection(s)
I20260929 02:51:54.330726 23179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/data/info.pb
I20260929 02:51:54.334820 23246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:54.339636 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 23179
I20260929 02:51:54.339847 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-0/wal/instance
I20260929 02:51:54.340426 388 external_mini_cluster.cc:1428] Setting key e433e010ca1143b1df7040f540402d90
I20260929 02:51:54.343114 23246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:54 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
I20260929 02:51:54.367484 23248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.346854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.0.97.62:38433)
0929 02:51:54.347797 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:54.347819 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:54.348290 (+ 471us) server_negotiation.cc:407] Connection header received
0929 02:51:54.350533 (+ 2243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:54.350539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:54.350820 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:54.351048 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:54.352398 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.353669 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.355114 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.355694 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.355916 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:54.359479 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:54.359488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:54.359502 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:54.359537 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:54.362562 (+ 3025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:54.363349 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:54.363366 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:54.363368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:54.363525 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:54.363936 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:54.363943 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:54.363948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:54.364601 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:54.364850 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:54.365394 (+ 544us) server_negotiation.cc:299] Negotiation successful
0929 02:51:54.365861 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":654,"thread_start_us":430,"threads_started":1}
I20260929 02:51:54.367506 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.346854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38433 (local address 127.0.0.1:53666)
0929 02:51:54.348093 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.348141 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:54.350096 (+ 1955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:54.351055 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:54.351061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:54.351822 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:54.352165 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:54.352183 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.353838 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.353845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:54.354964 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:54.354976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.355935 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.355942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:54.356113 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:54.356904 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:54.356932 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:54.359235 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:54.362720 (+ 3485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:54.362735 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:54.362750 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:54.363175 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:54.363645 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:54.363650 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:54.363654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:54.363823 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:54.364765 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:54.364774 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:54.365000 (+ 226us) client_negotiation.cc:769] Sending connection context
0929 02:51:54.365274 (+ 274us) client_negotiation.cc:240] Negotiation successful
0929 02:51:54.365983 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"mutex_wait_us":209,"thread_start_us":785,"threads_started":1}
I20260929 02:51:54.371474 23246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:51:54.372987 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.372420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59694)
0929 02:51:54.372725 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.372852 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260929 02:51:54.373551 23198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
I20260929 02:51:54.375362 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.374953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59706)
0929 02:51:54.375122 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.375206 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:51:54.375612 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36195: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260929 02:51:54.400465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36195
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38433,127.0.97.61:36195,127.0.97.60:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:51:54.424948 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } attempt: 1
I20260929 02:51:54.426241 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.425722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59720)
0929 02:51:54.425951 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.426076 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 02:51:54.427366 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.426985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59724)
0929 02:51:54.427131 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.427213 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260929 02:51:54.427569 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36195: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
I20260929 02:51:54.495868 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } attempt: 2
I20260929 02:51:54.497301 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.496699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59738)
0929 02:51:54.497005 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.497118 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 02:51:54.498584 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.498117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59750)
0929 02:51:54.498296 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.498401 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260929 02:51:54.498864 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36195: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
I20260929 02:51:54.590180 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } attempt: 3
I20260929 02:51:54.591459 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.590928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59754)
0929 02:51:54.591170 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.591289 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260929 02:51:54.592698 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.592185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59768)
0929 02:51:54.592354 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.592452 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 02:51:54.592983 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36195: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
I20260929 02:51:54.736266 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } attempt: 4
I20260929 02:51:54.737625 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.736996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59778)
0929 02:51:54.737344 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.737453 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260929 02:51:54.738799 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.738359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59780)
0929 02:51:54.738537 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.738636 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 02:51:54.739027 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:36195: Network error: Client connection negotiation failed: client connection to 127.0.97.61:36195: connect: Connection refused (error 111)
W20260929 02:51:54.747881 23253 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:51:54.748222 23253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:54.748296 23253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:54.748378 23253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:54.755822 23253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:54.755918 23253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:54.755978 23253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:54.756021 23253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:54.756084 23253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:54.766446 23253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38433,127.0.97.61:36195,127.0.97.60:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:54.767812 23253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:54.769160 23253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:54.781450 23258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:54.781497 23259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:54.781565 23261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:54.782366 23253 server_base.cc:1034] running on GCE node
I20260929 02:51:54.783238 23253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:54.784750 23253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:54.786032 23253 hybrid_clock.cc:648] HybridClock initialized: now 1790650314785930 us; error 68 us; skew 500 ppm
I20260929 02:51:54.786432 23253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:54 dist-test-slave-6tjh krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:54.794003 23253 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260929 02:51:54.795670 23253 webserver.cc:460] Webserver started at http://127.0.97.61:45973/ using document root <none> and password file <none>
I20260929 02:51:54.796506 23253 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:54.796664 23253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:54.796983 23253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:54.800047 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data/instance:
uuid: "ebc1d3bc99a844c68ff94b25008d72b0"
format_stamp: "Formatted at 2026-09-29 02:51:54 on dist-test-slave-6tjh"
server_key: "9015143c2a80646d10d18f399180e87d"
server_key_iv: "c76549d543ec0c1070cbce3f795d275b"
server_key_version: "encryptionkey@0"
I20260929 02:51:54.801017 23253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/wal/instance:
uuid: "ebc1d3bc99a844c68ff94b25008d72b0"
format_stamp: "Formatted at 2026-09-29 02:51:54 on dist-test-slave-6tjh"
server_key: "9015143c2a80646d10d18f399180e87d"
server_key_iv: "c76549d543ec0c1070cbce3f795d275b"
server_key_version: "encryptionkey@0"
I20260929 02:51:54.806459 23253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:51:54.810127 23268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:54.811584 23253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:51:54.811730 23253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/wal
uuid: "ebc1d3bc99a844c68ff94b25008d72b0"
format_stamp: "Formatted at 2026-09-29 02:51:54 on dist-test-slave-6tjh"
server_key: "9015143c2a80646d10d18f399180e87d"
server_key_iv: "c76549d543ec0c1070cbce3f795d275b"
server_key_version: "encryptionkey@0"
I20260929 02:51:54.811913 23253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:54.835028 23253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:54.839735 23253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:54.839969 23253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:54.859176 23253 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36195
I20260929 02:51:54.859220 23319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36195 every 8 connection(s)
I20260929 02:51:54.860683 23253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/data/info.pb
I20260929 02:51:54.864634 23320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:54.867290 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 23253
I20260929 02:51:54.867475 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-1/wal/instance
I20260929 02:51:54.867915 388 external_mini_cluster.cc:1428] Setting key ba3f3e1600aa4e473afba513bbaac257
I20260929 02:51:54.872124 23320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:54 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
I20260929 02:51:54.894831 23322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.874950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.0.97.62:38433)
0929 02:51:54.875468 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:54.875473 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:54.876675 (+ 1202us) server_negotiation.cc:407] Connection header received
0929 02:51:54.878649 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:54.878658 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:54.878768 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:54.878908 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:54.880526 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.881707 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.883274 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.884035 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.884193 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:54.887715 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:54.887725 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:54.887733 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:54.887799 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:54.891561 (+ 3762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:54.892404 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:54.892413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:54.892417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:54.892635 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:54.893040 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:54.893043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:54.893045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:54.893326 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:54.893602 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:54.894257 (+ 655us) server_negotiation.cc:299] Negotiation successful
0929 02:51:54.894531 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260929 02:51:54.895851 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.875629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38433 (local address 127.0.0.1:53678)
0929 02:51:54.876451 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.876502 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:54.878333 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:54.879118 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:54.879127 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:54.879674 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:54.880274 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:54.880292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.881875 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.881879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:54.883107 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:54.883120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.884248 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.884252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:54.884459 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:54.885242 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:54.885265 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:54.887448 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:54.891767 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:54.891775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:54.891789 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:54.892266 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:54.892754 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:54.892758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:54.892761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:54.892940 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:54.893531 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:54.893539 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:54.893916 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 02:51:54.894154 (+ 238us) client_negotiation.cc:240] Negotiation successful
0929 02:51:54.894560 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":358,"threads_started":1}
I20260929 02:51:54.897869 23320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:54 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:51:54.916745 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.898758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59786)
0929 02:51:54.899087 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.899107 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:54.899261 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:54.900432 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:54.900436 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:54.900887 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:54.901028 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:54.901038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.903149 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.903154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:54.904123 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:54.904137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.905266 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.905280 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:54.905524 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:54.906308 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:54.906338 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:54.909064 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:54.913996 (+ 4932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:54.914000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:54.914003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:54.914405 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:54.914908 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:54.914912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:54.914917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:54.915065 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:54.915904 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:54.915908 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:54.916086 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 02:51:54.916205 (+ 119us) client_negotiation.cc:240] Negotiation successful
0929 02:51:54.916419 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260929 02:51:54.917519 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.898957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.0.97.61:36195)
0929 02:51:54.899774 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:54.899784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:54.899818 (+ 34us) server_negotiation.cc:407] Connection header received
0929 02:51:54.899876 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:54.899886 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:54.900248 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:54.900408 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:54.901221 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.902955 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.904280 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:54.905096 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:54.905209 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:54.909335 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:54.909355 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:54.909378 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:54.909430 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:54.913825 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:54.914553 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:54.914566 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:54.914572 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:54.914761 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:54.915154 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:54.915162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:54.915169 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:54.915723 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:54.915975 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:54.916439 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 02:51:54.916797 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":447,"threads_started":1}
I20260929 02:51:54.920027 23320 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:51:54.921160 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.920727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57496)
0929 02:51:54.920900 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.921024 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:51:54.921446 23271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
I20260929 02:51:54.923053 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.922705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57500)
0929 02:51:54.922877 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.922951 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 02:51:54.923269 23320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260929 02:51:54.934065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
/tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45663
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38433,127.0.97.61:36195,127.0.97.60:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:51:54.972764 23320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 1
I20260929 02:51:54.974151 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.973513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57506)
0929 02:51:54.973709 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.973817 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 02:51:54.975289 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:54.974819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57512)
0929 02:51:54.974984 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:54.975114 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260929 02:51:54.975848 23320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
I20260929 02:51:55.038300 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } attempt: 5
I20260929 02:51:55.044091 23320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 2
I20260929 02:51:55.045173 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.044696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57516)
0929 02:51:55.044894 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.044994 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 02:51:55.046254 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.045889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57528)
0929 02:51:55.046037 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.046119 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:51:55.046559 23320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:51:55.053188 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.039195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.0.97.61:36195)
0929 02:51:55.039408 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.039412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.039432 (+ 20us) server_negotiation.cc:407] Connection header received
0929 02:51:55.039663 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.039667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.039763 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.039843 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.040888 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.041580 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.043462 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.044053 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.044200 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.047917 (+ 3717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.047927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.047931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.047961 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.050583 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.051268 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.051278 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.051280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.051393 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.051773 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.051777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.051780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.052065 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.052223 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.052631 (+ 408us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.052870 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 02:51:55.053210 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.039087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59794)
0929 02:51:55.039310 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.039335 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:55.039491 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:55.039926 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:55.039931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:55.040449 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:55.040732 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.040746 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.041741 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.041750 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.043299 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.043319 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.044243 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.044249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.044659 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.045399 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:55.045421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:55.047724 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:55.050726 (+ 3002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.050734 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.050738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.051128 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.051507 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.051512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.051515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.051653 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.052187 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:55.052192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:55.052347 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 02:51:55.052500 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 02:51:55.052849 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 02:51:55.054167 23246 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:51:55.055177 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.054815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57536)
0929 02:51:55.054967 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.055044 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 02:51:55.056143 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.055830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57540)
0929 02:51:55.055977 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.056053 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 02:51:55.056337 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
I20260929 02:51:55.107590 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 1
I20260929 02:51:55.109025 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.108360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57546)
0929 02:51:55.108678 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.108793 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 02:51:55.110334 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.109913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57548)
0929 02:51:55.110138 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.110215 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260929 02:51:55.110553 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
I20260929 02:51:55.137810 23320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 3
I20260929 02:51:55.139181 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.138614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57558)
0929 02:51:55.138869 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.138981 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260929 02:51:55.140442 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.139932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57568)
0929 02:51:55.140185 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.140285 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260929 02:51:55.140738 23320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
I20260929 02:51:55.178771 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 2
I20260929 02:51:55.179943 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.179453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57576)
0929 02:51:55.179673 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.179778 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 02:51:55.181212 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.180824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57580)
0929 02:51:55.181006 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.181090 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 02:51:55.181492 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
I20260929 02:51:55.284020 23320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 4
I20260929 02:51:55.285442 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.284848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57594)
0929 02:51:55.285130 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.285246 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 02:51:55.286613 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.286214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57604)
0929 02:51:55.286380 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.286471 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 02:51:55.286852 23320 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
I20260929 02:51:55.287747 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 3
I20260929 02:51:55.288885 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.288391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57614)
0929 02:51:55.288655 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.288738 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260929 02:51:55.290138 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.289564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57624)
0929 02:51:55.289838 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.289955 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260929 02:51:55.290409 23246 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:45663: Network error: Client connection negotiation failed: client connection to 127.0.97.60:45663: connect: Connection refused (error 111)
W20260929 02:51:55.297674 23328 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 02:51:55.298046 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:51:55.298172 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:51:55.298309 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 02:51:55.305826 23328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:51:55.305961 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 02:51:55.306085 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:51:55.306169 23328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:51:55.306275 23328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:51:55.317059 23328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38433,127.0.97.61:36195,127.0.97.60:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 01:55:41 UTC on ccycloud.quasar-gyuyoq.root.comops.site
build id 4679195
ASAN enabled
I20260929 02:51:55.318444 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:51:55.319897 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 02:51:55.332875 23333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:55.333004 23334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:51:55.333715 23336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:51:55.334456 23328 server_base.cc:1034] running on GCE node
I20260929 02:51:55.335346 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:51:55.336880 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:51:55.338150 23328 hybrid_clock.cc:648] HybridClock initialized: now 1790650315338073 us; error 52 us; skew 500 ppm
I20260929 02:51:55.338554 23328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:51:55.347765 23328 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260929 02:51:55.350085 23328 webserver.cc:460] Webserver started at http://127.0.97.60:38171/ using document root <none> and password file <none>
I20260929 02:51:55.350868 23328 fs_manager.cc:359] Metadata directory not provided
I20260929 02:51:55.351012 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:51:55.351364 23328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:51:55.354598 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data/instance:
uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c"
format_stamp: "Formatted at 2026-09-29 02:51:55 on dist-test-slave-6tjh"
server_key: "84ab7ce1055495222d2de4b87dd7a4a3"
server_key_iv: "9ae21e7dae4596521f17aedd0b1d4c28"
server_key_version: "encryptionkey@0"
I20260929 02:51:55.355441 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/wal/instance:
uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c"
format_stamp: "Formatted at 2026-09-29 02:51:55 on dist-test-slave-6tjh"
server_key: "84ab7ce1055495222d2de4b87dd7a4a3"
server_key_iv: "9ae21e7dae4596521f17aedd0b1d4c28"
server_key_version: "encryptionkey@0"
I20260929 02:51:55.361155 23328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 02:51:55.365137 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:51:55.366694 23328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 02:51:55.366921 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/wal
uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c"
format_stamp: "Formatted at 2026-09-29 02:51:55 on dist-test-slave-6tjh"
server_key: "84ab7ce1055495222d2de4b87dd7a4a3"
server_key_iv: "9ae21e7dae4596521f17aedd0b1d4c28"
server_key_version: "encryptionkey@0"
I20260929 02:51:55.367180 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:51:55.384335 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 02:51:55.388484 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:51:55.388815 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:51:55.408146 23328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:45663
I20260929 02:51:55.408167 23394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:45663 every 8 connection(s)
I20260929 02:51:55.409659 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/data/info.pb
I20260929 02:51:55.411419 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu as pid 23328
I20260929 02:51:55.411607 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskMdzdaQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790649949495923-388-0/minicluster-data/master-2/wal/instance
I20260929 02:51:55.412140 388 external_mini_cluster.cc:1428] Setting key ae8156cb2f7ebf080707ce9257fd8e89
I20260929 02:51:55.414601 23395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:51:55.423075 23395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260929 02:51:55.437781 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.414074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57632 (local address 127.0.97.60:45663)
0929 02:51:55.414898 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.414911 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.414953 (+ 42us) server_negotiation.cc:407] Connection header received
0929 02:51:55.416875 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.416897 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.417326 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.417663 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.419460 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.420911 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.422668 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.423395 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.423550 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.428812 (+ 5262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.428819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.428833 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.428882 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.432864 (+ 3982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.433816 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.433824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.433831 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.434068 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.434584 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.434588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.434591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.435329 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.435633 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.436079 (+ 446us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.436498 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":352,"threads_started":1}
I20260929 02:51:55.442880 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.426086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:38433 (local address 127.0.0.1:53694)
0929 02:51:55.426911 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.426963 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:55.427143 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:55.427611 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:55.427619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:55.428501 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:55.428905 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.428917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.430388 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.430392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.431691 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.431712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.432855 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.432860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.433043 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.433725 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:55.433753 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:55.436377 (+ 2624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:55.439633 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.439642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.439648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.440127 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.440574 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.440579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.440583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.440762 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.441668 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:55.441677 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:55.441927 (+ 250us) client_negotiation.cc:769] Sending connection context
0929 02:51:55.442184 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 02:51:55.442575 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":428,"threads_started":1}
I20260929 02:51:55.443010 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.426008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53694 (local address 127.0.97.62:38433)
0929 02:51:55.426551 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.426559 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.427085 (+ 526us) server_negotiation.cc:407] Connection header received
0929 02:51:55.427296 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.427303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.427409 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.427500 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.429076 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.430200 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.431889 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.432657 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.432823 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.436614 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.436619 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.436623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.436666 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.439471 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.440267 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.440273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.440277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.440431 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.440919 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.440924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.440926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.441393 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.441574 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.442297 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.442553 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":291,"threads_started":1}
I20260929 02:51:55.444777 23395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:51:55.458563 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.441453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.0.97.62:38433)
0929 02:51:55.442134 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.442141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.442167 (+ 26us) server_negotiation.cc:407] Connection header received
0929 02:51:55.442217 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.442221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.442298 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.442400 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.443973 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.445097 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.446419 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.446861 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.447000 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.451964 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.451969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.451972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.452006 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.455657 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.456385 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.456393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.456398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.456560 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.456977 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.456980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.456982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.457293 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.457575 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.457947 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.458168 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":295,"threads_started":1}
I20260929 02:51:55.459324 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.445494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36195 (local address 127.0.0.1:59798)
0929 02:51:55.445650 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.445671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:55.445783 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:55.446090 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:55.446095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:55.446458 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:55.446597 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.446605 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.447615 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.447639 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.449220 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.449238 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.449944 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.449953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.450189 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.450739 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:55.450755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:55.453208 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:55.456668 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.456672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.456676 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.457153 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.457567 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.457572 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.457576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.457750 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.458408 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:55.458414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:55.458564 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 02:51:55.458708 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 02:51:55.458990 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 02:51:55.459834 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.445597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.0.97.61:36195)
0929 02:51:55.445801 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.445808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.445832 (+ 24us) server_negotiation.cc:407] Connection header received
0929 02:51:55.445891 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.445896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.445966 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.446068 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.446750 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.447458 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.449376 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.449788 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.449880 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.453401 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.453408 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.453413 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.453457 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.456475 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.457268 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.457274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.457277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.457435 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.457865 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.457871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.457874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.458187 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.458365 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.459052 (+ 687us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.459264 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:51:55.460822 23395 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:51:55.467658 23246 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 4
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260929 02:51:55.476897 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.461502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57640 (local address 127.0.97.60:45663)
0929 02:51:55.461658 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.461662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.461678 (+ 16us) server_negotiation.cc:407] Connection header received
0929 02:51:55.461871 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.461879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.461947 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.462009 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.462772 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.463601 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.465148 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.465726 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.465815 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.470251 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.470256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.470259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.470289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.473823 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.474712 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.474717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.474723 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.474856 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.475276 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.475282 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.475285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.475656 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.475821 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.476201 (+ 380us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.476450 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 02:51:55.477020 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.461484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57640)
0929 02:51:55.461634 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.461647 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:55.461721 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:55.462190 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:55.462193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:55.462521 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:55.462639 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.462646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.463732 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.463737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.465004 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.465033 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.466205 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.466213 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.466424 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.466991 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:55.467017 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:55.470043 (+ 3026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:55.473977 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.473982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.473986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.474593 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.474976 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.474981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.474984 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.475151 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.475783 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:55.475787 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:55.475949 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 02:51:55.476124 (+ 175us) client_negotiation.cc:240] Negotiation successful
0929 02:51:55.476671 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":169,"spinlock_wait_cycles":30080}
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 02:51:55.478369 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.460196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59812 (local address 127.0.97.61:36195)
0929 02:51:55.460347 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.460351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.460365 (+ 14us) server_negotiation.cc:407] Connection header received
0929 02:51:55.460536 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.460572 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.460650 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.460771 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.462274 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.463173 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.464778 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.465280 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.465404 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.470860 (+ 5456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.470866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.470869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.470902 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.474456 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.475414 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.475420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.475424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.475569 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.475984 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.475989 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.475993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.476341 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.476656 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.477806 (+ 1150us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.478048 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 02:51:55.482580 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: Bootstrap starting.
I20260929 02:51:55.483312 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 02:51:55.486754 23395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:55.488242 23395 log.cc:824] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:55.486624 23221 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "4c063aef7c6041a495e2bb1a3da85404"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53706
I20260929 02:51:55.491510 23395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: No bootstrap required, opened a new log
I20260929 02:51:55.491955 23249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.468375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57646)
0929 02:51:55.468738 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.468759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:55.468884 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:55.469825 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:55.469832 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:55.470382 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:55.470603 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.470615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.472296 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.472302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.473506 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.473520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.474424 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.474430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.474663 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.475276 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:55.475300 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:55.485078 (+ 9778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:55.488674 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.488680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.488684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.489362 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.489902 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.489907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.489912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.490100 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.490881 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:55.490885 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:55.491060 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 02:51:55.491221 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 02:51:55.491551 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227}
I20260929 02:51:55.492009 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.468593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.0.97.60:45663)
0929 02:51:55.469442 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.469452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.469475 (+ 23us) server_negotiation.cc:407] Connection header received
0929 02:51:55.469549 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.469555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.469632 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.469754 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.470827 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.472103 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.473667 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.474255 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.474401 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.485297 (+ 10896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.485303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.485306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.485340 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.488455 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.489544 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.489551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.489554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.489741 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.490249 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.490254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.490259 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.490603 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.490786 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.491331 (+ 545us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.491548 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":532,"threads_started":1}
I20260929 02:51:55.497107 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404: Bootstrap starting.
I20260929 02:51:55.500262 23246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:55.501328 23246 log.cc:824] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:55.502310 23395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.502758 23395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d3a0b94f9ac4f7ba1104e14adc2026c, State: Initialized, Role: FOLLOWER
I20260929 02:51:55.503286 23395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.504189 23246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404: No bootstrap required, opened a new log
I20260929 02:51:55.504256 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.504716 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:55.505524 23395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:55.512063 23246 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.512490 23246 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c063aef7c6041a495e2bb1a3da85404, State: Initialized, Role: FOLLOWER
I20260929 02:51:55.513036 23246 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.514273 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.514962 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:55.515559 23246 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:55.515599 23221 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 02:51:55.515786 23221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:55.520097 23221 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.524899 23370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c063aef7c6041a495e2bb1a3da85404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c"
I20260929 02:51:55.525482 23221 leader_election.cc:288] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [CANDIDATE]: Term 1 election: Requested vote from peers ebc1d3bc99a844c68ff94b25008d72b0 (127.0.97.61:36195), 5d3a0b94f9ac4f7ba1104e14adc2026c (127.0.97.60:45663)
I20260929 02:51:55.525651 23370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:55.530355 23370 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c063aef7c6041a495e2bb1a3da85404 in term 1.
I20260929 02:51:55.531296 23197 leader_election.cc:302] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c063aef7c6041a495e2bb1a3da85404, 5d3a0b94f9ac4f7ba1104e14adc2026c; no voters:
I20260929 02:51:55.534571 23417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:51:55.514057 23328 master_runner.cc:186] Error getting master registration for 127.0.97.62:38433: OK, instance_id { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" instance_seqno: 1790650314323300 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:51:55.536772 23328 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 02:51:55.539875 23416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:51:55.540112 23416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:51:55.540992 23417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [term 1 LEADER]: Becoming Leader. State: Replica: 4c063aef7c6041a495e2bb1a3da85404, State: Running, Role: LEADER
I20260929 02:51:55.545512 23417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.552696 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c063aef7c6041a495e2bb1a3da85404. Latest consensus state: current_term: 1 leader_uuid: "4c063aef7c6041a495e2bb1a3da85404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.553203 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:55.554500 23436 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 02:51:55.562563 23200 tablet.cc:2369] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:55.582178 23436 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 02:51:55.586021 23320 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } attempt: 5
I20260929 02:51:55.578104 23179 master_runner.cc:186] Error getting master registration for 127.0.97.61:36195: OK, instance_id { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" instance_seqno: 1790650314853775 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
Sep 29 02:51:55 dist-test-slave-6tjh krb5kdc[23166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790650314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260929 02:51:55.604265 23179 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:51:55.605540 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.587437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.0.97.60:45663)
0929 02:51:55.587657 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.587663 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.587682 (+ 19us) server_negotiation.cc:407] Connection header received
0929 02:51:55.587731 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.587736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.587813 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.587916 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.589016 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.590119 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.591833 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.592538 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.592707 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.597247 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.597254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.597259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.597320 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.600994 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.601923 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.601930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.601934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.602122 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.602789 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.602794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.602798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.603204 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.603336 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.604379 (+ 1043us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.604841 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 02:51:55.606971 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.548433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.0.97.62:38433)
0929 02:51:55.548694 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.548701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.553215 (+ 4514us) server_negotiation.cc:407] Connection header received
0929 02:51:55.554824 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.554831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.554946 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.555025 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.557239 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.558383 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.568635 (+ 10252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.569382 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.569511 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.593016 (+ 23505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.593025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.593029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.593073 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.597007 (+ 3934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.597970 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.597978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.597983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.598171 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.605000 (+ 6829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.605006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.605010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.605429 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.605708 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.606354 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.606605 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:51:55.607088 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.586696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:45663 (local address 127.0.0.1:57666)
0929 02:51:55.587094 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0929 02:51:55.587117 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 02:51:55.587231 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 02:51:55.588094 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 02:51:55.588100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 02:51:55.588629 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 02:51:55.588842 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.588855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.590262 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.590268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.591691 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 02:51:55.591708 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.592735 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.592742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 02:51:55.592945 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.593602 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 02:51:55.593625 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 02:51:55.597004 (+ 3379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 02:51:55.601203 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.601209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.601214 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.601787 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.602323 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 02:51:55.602329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 02:51:55.602333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 02:51:55.602600 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 02:51:55.603478 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 02:51:55.603504 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 02:51:55.603762 (+ 258us) client_negotiation.cc:769] Sending connection context
0929 02:51:55.605971 (+ 2209us) client_negotiation.cc:240] Negotiation successful
0929 02:51:55.606441 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260929 02:51:55.613845 23320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0: Bootstrap starting.
I20260929 02:51:55.617241 23370 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [term 1 FOLLOWER]: Refusing update from remote peer 4c063aef7c6041a495e2bb1a3da85404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:55.618054 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.560043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57654 (local address 127.0.97.60:45663)
0929 02:51:55.563751 (+ 3708us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.563760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.578544 (+ 14784us) server_negotiation.cc:407] Connection header received
0929 02:51:55.579047 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.579056 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.579173 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.579252 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.583336 (+ 4084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.584499 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.586522 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.587178 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.587298 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.608025 (+ 20727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.608034 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.608039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.608076 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.613211 (+ 5135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.614745 (+ 1534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.614751 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.614754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.614936 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.615492 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.615497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.615500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.615874 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.616129 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.617400 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.617654 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 02:51:55.620136 23320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0: Neither blocks nor log segments found. Creating new log.
I20260929 02:51:55.620055 23424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:55.622287 23320 log.cc:824] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0: Log is configured to *not* fsync() on all Append() calls
I20260929 02:51:55.630503 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:55.559914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59818 (local address 127.0.97.61:36195)
0929 02:51:55.560129 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.560138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.574366 (+ 14228us) server_negotiation.cc:407] Connection header received
0929 02:51:55.574451 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.574457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.574550 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.574670 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.604607 (+ 29937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.605740 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.609436 (+ 3696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:55.610200 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:55.610300 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:55.621212 (+ 10912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 02:51:55.621219 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 02:51:55.621224 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 02:51:55.621268 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 02:51:55.625160 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.626200 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.626205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.626208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.626360 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 02:51:55.627650 (+ 1290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 02:51:55.627655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 02:51:55.627658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 02:51:55.628054 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 02:51:55.628507 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:55.629453 (+ 946us) server_negotiation.cc:299] Negotiation successful
0929 02:51:55.629913 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 02:51:55.633605 23320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0: No bootstrap required, opened a new log
W20260929 02:51:55.640987 23349 tablet.cc:2369] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:51:55.651283 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c063aef7c6041a495e2bb1a3da85404. Latest consensus state: current_term: 1 leader_uuid: "4c063aef7c6041a495e2bb1a3da85404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.651619 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:55.652310 23441 mvcc.cc:205] Tried to move back new op lower bound from 7334503692741742592 to 7334503692492337152. Current Snapshot: MvccSnapshot[applied={T|T < 7334503692741742592}]
I20260929 02:51:55.654099 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c063aef7c6041a495e2bb1a3da85404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.654327 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:55.655660 23320 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.656575 23320 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebc1d3bc99a844c68ff94b25008d72b0, State: Initialized, Role: FOLLOWER
I20260929 02:51:55.657646 23320 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } }
I20260929 02:51:55.665468 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.666309 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:55.667773 23295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4c063aef7c6041a495e2bb1a3da85404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ebc1d3bc99a844c68ff94b25008d72b0"
I20260929 02:51:55.668110 23294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:51:55.668485 23320 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:51:55.671329 23424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c063aef7c6041a495e2bb1a3da85404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.671706 23424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:55.673712 23436 catalog_manager.cc:1347] Generated new cluster ID: 2e2b9e1389364dfea23336538bd77e7e
I20260929 02:51:55.673810 23436 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 02:51:55.679286 23294 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [term 1 FOLLOWER]: Refusing update from remote peer 4c063aef7c6041a495e2bb1a3da85404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:51:55.686661 23295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c063aef7c6041a495e2bb1a3da85404 in term 1.
I20260929 02:51:55.689247 23417 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:51:55.734901 23447 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c063aef7c6041a495e2bb1a3da85404. Latest consensus state: current_term: 1 leader_uuid: "4c063aef7c6041a495e2bb1a3da85404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.735285 23447 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:55.735621 23458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4c063aef7c6041a495e2bb1a3da85404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.736003 23458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:51:55.755577 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4c063aef7c6041a495e2bb1a3da85404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c063aef7c6041a495e2bb1a3da85404" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38433 } } peers { permanent_uuid: "ebc1d3bc99a844c68ff94b25008d72b0" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36195 } } peers { permanent_uuid: "5d3a0b94f9ac4f7ba1104e14adc2026c" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 45663 } } }
I20260929 02:51:55.756022 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404 [sys.catalog]: This master's current role is: LEADER
I20260929 02:51:55.759410 23436 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 02:51:55.770562 23436 catalog_manager.cc:1504] Loading token signing keys...
I20260929 02:51:55.788142 23462 catalog_manager.cc:1259] Loaded cluster ID: 2e2b9e1389364dfea23336538bd77e7e
I20260929 02:51:55.788352 23462 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0: loading cluster ID for follower catalog manager: success
I20260929 02:51:55.793179 23436 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c063aef7c6041a495e2bb1a3da85404: Generated new TSK 0
I20260929 02:51:55.796416 23436 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 02:51:55.798940 23462 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0: acquiring CA information for follower catalog manager: success
I20260929 02:51:55.801077 23462 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ebc1d3bc99a844c68ff94b25008d72b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:51:55.962023 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:55.949710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53728 (local address 127.0.97.62:38433)
0929 02:51:55.949961 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.949971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.958447 (+ 8476us) server_negotiation.cc:407] Connection header received
0929 02:51:55.959565 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.959573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.959680 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.959761 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.961702 (+ 1941us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53728: BlockingRecv error: recv got EOF from 127.0.0.1:53728 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260929 02:51:55.966301 23325 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:55.953565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.0.97.61:36195)
0929 02:51:55.953805 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.953813 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.963002 (+ 9189us) server_negotiation.cc:407] Connection header received
0929 02:51:55.964823 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.964834 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.964952 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.965030 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.966014 (+ 984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59826: BlockingRecv error: recv got EOF from 127.0.0.1:59826 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260929 02:51:55.968626 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:55.963828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57682 (local address 127.0.97.60:45663)
0929 02:51:55.963997 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:55.964004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:55.964524 (+ 520us) server_negotiation.cc:407] Connection header received
0929 02:51:55.966945 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:55.966951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:55.967050 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:55.967134 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:55.968290 (+ 1156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57682: BlockingRecv error: recv got EOF from 127.0.0.1:57682 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 02:51:56.021366 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.004699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.0.97.61:36195)
0929 02:51:56.004900 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.004907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.011945 (+ 7038us) server_negotiation.cc:407] Connection header received
0929 02:51:56.012631 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.012638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.012854 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.012991 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.013750 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.015145 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.017816 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.018565 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.018686 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.019605 (+ 919us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.020421 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.020594 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.020874 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.020993 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 02:51:56.068138 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.057771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.0.97.60:45663)
0929 02:51:56.058005 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.058013 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.065639 (+ 7626us) server_negotiation.cc:407] Connection header received
0929 02:51:56.066296 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.066305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.066446 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.066570 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.067856 (+ 1286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57698: BlockingRecv error: recv got EOF from 127.0.0.1:57698 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 02:51:56.115288 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.099825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.0.97.61:36195)
0929 02:51:56.100046 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.100054 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.107122 (+ 7068us) server_negotiation.cc:407] Connection header received
0929 02:51:56.107731 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.107739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.107881 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.107971 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.108883 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.110199 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.112567 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.113290 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.113416 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.114164 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.114349 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.114434 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.114730 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.114835 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 02:51:56.161144 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.150986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.0.97.60:45663)
0929 02:51:56.151208 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.151216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.158848 (+ 7632us) server_negotiation.cc:407] Connection header received
0929 02:51:56.159427 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.159435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.159580 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.159669 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.160879 (+ 1210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57710: BlockingRecv error: recv got EOF from 127.0.0.1:57710 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 02:51:56.214438 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.199268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.0.97.61:36195)
0929 02:51:56.199477 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.199485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.206528 (+ 7043us) server_negotiation.cc:407] Connection header received
0929 02:51:56.207095 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.207104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.207212 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.207283 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.208064 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.209504 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.211857 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.212712 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.212820 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.213473 (+ 653us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.213649 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.213717 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.213954 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.214058 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 02:51:56.255369 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.245413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.0.97.60:45663)
0929 02:51:56.245684 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.245691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.253351 (+ 7660us) server_negotiation.cc:407] Connection header received
0929 02:51:56.253887 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.253893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.253990 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.254069 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.255120 (+ 1051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57726: BlockingRecv error: recv got EOF from 127.0.0.1:57726 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
I20260929 02:51:56.269809 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.263594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.97.61:36195)
0929 02:51:56.263933 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.263943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.264487 (+ 544us) server_negotiation.cc:407] Connection header received
0929 02:51:56.264789 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.264800 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.264873 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.265005 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.265459 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.266348 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.267837 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.268376 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.268568 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.269041 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.269168 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.269277 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.269411 (+ 134us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.269523 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
W20260929 02:51:56.278451 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.276251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.0.97.60:45663)
0929 02:51:56.276542 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.276576 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.276929 (+ 353us) server_negotiation.cc:407] Connection header received
0929 02:51:56.277089 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.277095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.277217 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.277351 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.278190 (+ 839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57730: BlockingRecv error: recv got EOF from 127.0.0.1:57730 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
I20260929 02:51:56.295763 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.289912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.0.97.61:36195)
0929 02:51:56.290114 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.290123 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.290537 (+ 414us) server_negotiation.cc:407] Connection header received
0929 02:51:56.290774 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.290780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.290857 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.290980 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.291402 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.292193 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.293714 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.294143 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.294296 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.294696 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.294820 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.294918 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.295100 (+ 182us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.295457 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260929 02:51:56.305297 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.303336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.0.97.60:45663)
0929 02:51:56.303533 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.303540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.303857 (+ 317us) server_negotiation.cc:407] Connection header received
0929 02:51:56.304024 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.304029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.304126 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.304249 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.305050 (+ 801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57740: BlockingRecv error: recv got EOF from 127.0.0.1:57740 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 02:51:56.331449 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.325161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59882 (local address 127.0.97.61:36195)
0929 02:51:56.325380 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.325389 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.325766 (+ 377us) server_negotiation.cc:407] Connection header received
0929 02:51:56.325994 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.326005 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.326102 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.326202 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.326610 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.327409 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.329191 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.329940 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.330081 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.330607 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.330778 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.330866 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.331066 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.331163 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 02:51:56.351367 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.349335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57750 (local address 127.0.97.60:45663)
0929 02:51:56.349521 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.349528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.349739 (+ 211us) server_negotiation.cc:407] Connection header received
0929 02:51:56.349941 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.349945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.350020 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.350116 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.351092 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57750: BlockingRecv error: recv got EOF from 127.0.0.1:57750 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 02:51:56.393579 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.387801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.0.97.61:36195)
0929 02:51:56.387988 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.387993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.388390 (+ 397us) server_negotiation.cc:407] Connection header received
0929 02:51:56.388645 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.388649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.388762 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.388853 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.389337 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.390059 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.391542 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.392014 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.392111 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.392601 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.392733 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.392819 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.393239 (+ 420us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.393314 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 02:51:56.402477 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.400357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57764 (local address 127.0.97.60:45663)
0929 02:51:56.400615 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.400627 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.401001 (+ 374us) server_negotiation.cc:407] Connection header received
0929 02:51:56.401211 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.401215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.401290 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.401389 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.402202 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57764: BlockingRecv error: recv got EOF from 127.0.0.1:57764 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 02:51:56.475596 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.470236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.0.97.61:36195)
0929 02:51:56.470416 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.470421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.470769 (+ 348us) server_negotiation.cc:407] Connection header received
0929 02:51:56.470973 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.470978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.471060 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.471207 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.471722 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.472504 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.473881 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.474322 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.474472 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.474869 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.474979 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.475080 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.475239 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.475321 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 02:51:56.483120 23402 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 02:51:56.481369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.0.97.60:45663)
0929 02:51:56.481543 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.481548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.481825 (+ 277us) server_negotiation.cc:407] Connection header received
0929 02:51:56.481972 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.481976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.482078 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.482171 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 02:51:56.482882 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57770: BlockingRecv error: recv got EOF from 127.0.0.1:57770 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 02:51:56.542404 23416 catalog_manager.cc:1259] Loaded cluster ID: 2e2b9e1389364dfea23336538bd77e7e
I20260929 02:51:56.542519 23416 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: loading cluster ID for follower catalog manager: success
I20260929 02:51:56.545408 23416 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: acquiring CA information for follower catalog manager: success
I20260929 02:51:56.547461 23416 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5d3a0b94f9ac4f7ba1104e14adc2026c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:51:56.619845 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.614357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.0.97.61:36195)
0929 02:51:56.614614 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.614621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.614958 (+ 337us) server_negotiation.cc:407] Connection header received
0929 02:51:56.615248 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.615253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.615335 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.615458 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.615913 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.616744 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.618113 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.618557 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.618697 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.619114 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.619225 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.619325 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.619480 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.619563 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 02:51:56.632124 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.625989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.0.97.60:45663)
0929 02:51:56.626276 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.626284 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.626541 (+ 257us) server_negotiation.cc:407] Connection header received
0929 02:51:56.626778 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.626782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.626945 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.627089 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.627505 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.628256 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.629769 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.630220 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.630370 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.630925 (+ 555us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.631461 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.631632 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.631734 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.631828 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 02:51:56.650725 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.639129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.0.97.61:36195)
0929 02:51:56.639403 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.639412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.644147 (+ 4735us) server_negotiation.cc:407] Connection header received
0929 02:51:56.644363 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.644372 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.644486 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.644633 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.645371 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.646550 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.648471 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.649160 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.649288 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.649876 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.650013 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.650105 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.650311 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.650402 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 02:51:56.651263 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.640392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.0.97.60:45663)
0929 02:51:56.640642 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.640650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.644884 (+ 4234us) server_negotiation.cc:407] Connection header received
0929 02:51:56.645182 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.645189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.645285 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.645385 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.646272 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.647410 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.649286 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.649894 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.650028 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.650529 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.650676 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.650741 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.650974 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.651059 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 02:51:56.652743 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 02:51:56.638925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53736 (local address 127.0.97.62:38433)
0929 02:51:56.645145 (+ 6220us) server_negotiation.cc:206] Beginning negotiation
0929 02:51:56.645152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 02:51:56.645169 (+ 17us) server_negotiation.cc:407] Connection header received
0929 02:51:56.645292 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 02:51:56.645297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 02:51:56.645514 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 02:51:56.645771 (+ 257us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 02:51:56.646478 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.647618 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.649619 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 02:51:56.650214 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 02:51:56.650843 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 02:51:56.651239 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 02:51:56.651798 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 02:51:56.651949 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 02:51:56.652333 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 02:51:56.652420 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6069,"thread_start_us":5969,"threads_started":1}
I20260929 02:51:56.657711 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 23179
I20260929 02:51:56.658768 23241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:56.773438 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 23179
I20260929 02:51:56.797731 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 23253
I20260929 02:51:56.798969 23315 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:56.906807 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 23253
I20260929 02:51:56.927376 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu pid 23328
I20260929 02:51:56.928789 23390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:51:57.038795 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskMdzdaQ/build/asan/bin/kudu with pid 23328
2026-09-29T02:51:57Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4563 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9040 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (367507 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST